Attachment 1450325 Details for Bug 1590143 – attaching heketi-pod logs

attaching heketi-pod logs

heketi.log (text/plain), 2.53 MB, created by Apeksha on 2018-06-12 06:26:38 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: Apeksha

Created: 2018-06-12 06:26:38 UTC

Size: 2.53 MB

patch

obsolete

>Heketi 7.0.0
>[heketi] INFO 2018/06/08 07:37:14 Loaded kubernetes executor
>[heketi] INFO 2018/06/08 07:37:14 Block: Auto Create Block Hosting Volume set to true
>[heketi] INFO 2018/06/08 07:37:14 Block: New Block Hosting Volume size 100 GB
>[heketi] INFO 2018/06/08 07:37:14 GlusterFS Application Loaded
>[heketi] INFO 2018/06/08 07:37:14 Started Node Health Cache Monitor
>Authorization loaded
>Listening on port 8080
>[negroni] Started POST /volumes
>[heketi] INFO 2018/06/08 07:37:20 Allocating brick set #0
>[negroni] Completed 202 Accepted in 21.131527ms
>[asynchttp] INFO 2018/06/08 07:37:20 asynchttp.go:288: Started job 0a3a0f4fc1906a402387b85f96b23c96
>[heketi] INFO 2018/06/08 07:37:20 Started async operation: Create Volume
>[negroni] Started GET /queue/0a3a0f4fc1906a402387b85f96b23c96
>[negroni] Completed 200 OK in 118.624Âµs
>[heketi] INFO 2018/06/08 07:37:20 Creating brick b9d88edeed17a4ff0395a5b36b4cd7e0
>[heketi] INFO 2018/06/08 07:37:20 Creating brick 4c6c93fd3801e871481fcd4fbf9b8ba8
>[heketi] INFO 2018/06/08 07:37:20 Creating brick 4481cc39f08c0bdcff489aab80756871
>[kubeexec] DEBUG 2018/06/08 07:37:20 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir -p /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_4481cc39f08c0bdcff489aab80756871
>Result: 
>[kubeexec] DEBUG 2018/06/08 07:37:20 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir -p /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_b9d88edeed17a4ff0395a5b36b4cd7e0
>Result: 
>[kubeexec] DEBUG 2018/06/08 07:37:20 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir -p /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_4c6c93fd3801e871481fcd4fbf9b8ba8
>Result: 
>[kubeexec] DEBUG 2018/06/08 07:37:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvcreate --autobackup=n --poolmetadatasize 53248K --chunksize 256K --size 10485760K --thin vg_d389f0278a774bd7443a09af960961d8/tp_4481cc39f08c0bdcff489aab80756871 --virtualsize 10485760K --name brick_4481cc39f08c0bdcff489aab80756871
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_4481cc39f08c0bdcff489aab80756871" created.
>[kubeexec] DEBUG 2018/06/08 07:37:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvcreate --autobackup=n --poolmetadatasize 53248K --chunksize 256K --size 10485760K --thin vg_96f1667f2f1ced2c5ef94772922be93b/tp_b9d88edeed17a4ff0395a5b36b4cd7e0 --virtualsize 10485760K --name brick_b9d88edeed17a4ff0395a5b36b4cd7e0
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_b9d88edeed17a4ff0395a5b36b4cd7e0" created.
>[kubeexec] DEBUG 2018/06/08 07:37:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvcreate --autobackup=n --poolmetadatasize 53248K --chunksize 256K --size 10485760K --thin vg_3a4297677881963e3f80124971d50eea/tp_4c6c93fd3801e871481fcd4fbf9b8ba8 --virtualsize 10485760K --name brick_4c6c93fd3801e871481fcd4fbf9b8ba8
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_4c6c93fd3801e871481fcd4fbf9b8ba8" created.
>[kubeexec] DEBUG 2018/06/08 07:37:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_4481cc39f08c0bdcff489aab80756871
>Result: meta-data=/dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_4481cc39f08c0bdcff489aab80756871 isize=512    agcount=16, agsize=163840 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=2621440, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 07:37:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_b9d88edeed17a4ff0395a5b36b4cd7e0
>Result: meta-data=/dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_b9d88edeed17a4ff0395a5b36b4cd7e0 isize=512    agcount=16, agsize=163840 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=2621440, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 07:37:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_4c6c93fd3801e871481fcd4fbf9b8ba8
>Result: meta-data=/dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_4c6c93fd3801e871481fcd4fbf9b8ba8 isize=512    agcount=16, agsize=163840 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=2621440, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 07:37:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: awk "BEGIN {print \"/dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_4481cc39f08c0bdcff489aab80756871 /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_4481cc39f08c0bdcff489aab80756871 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 07:37:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: awk "BEGIN {print \"/dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_b9d88edeed17a4ff0395a5b36b4cd7e0 /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_b9d88edeed17a4ff0395a5b36b4cd7e0 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 07:37:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: awk "BEGIN {print \"/dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_4c6c93fd3801e871481fcd4fbf9b8ba8 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_4c6c93fd3801e871481fcd4fbf9b8ba8 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[negroni] Started GET /queue/0a3a0f4fc1906a402387b85f96b23c96
>[negroni] Completed 200 OK in 133.427Âµs
>[kubeexec] DEBUG 2018/06/08 07:37:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_4481cc39f08c0bdcff489aab80756871 /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_4481cc39f08c0bdcff489aab80756871
>Result: 
>[kubeexec] DEBUG 2018/06/08 07:37:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_b9d88edeed17a4ff0395a5b36b4cd7e0 /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_b9d88edeed17a4ff0395a5b36b4cd7e0
>Result: 
>[kubeexec] DEBUG 2018/06/08 07:37:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_4c6c93fd3801e871481fcd4fbf9b8ba8 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_4c6c93fd3801e871481fcd4fbf9b8ba8
>Result: 
>[kubeexec] DEBUG 2018/06/08 07:37:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_4481cc39f08c0bdcff489aab80756871/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 07:37:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_b9d88edeed17a4ff0395a5b36b4cd7e0/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 07:37:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_4c6c93fd3801e871481fcd4fbf9b8ba8/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 07:37:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: chown :2001 /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_4481cc39f08c0bdcff489aab80756871/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 07:37:22 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: chown :2001 /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_b9d88edeed17a4ff0395a5b36b4cd7e0/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 07:37:22 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: chown :2001 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_4c6c93fd3801e871481fcd4fbf9b8ba8/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 07:37:22 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: chmod 2775 /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_4481cc39f08c0bdcff489aab80756871/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 07:37:22 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: chmod 2775 /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_b9d88edeed17a4ff0395a5b36b4cd7e0/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 07:37:22 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: chmod 2775 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_4c6c93fd3801e871481fcd4fbf9b8ba8/brick
>Result: 
>[cmdexec] INFO 2018/06/08 07:37:22 Creating volume vol_bcb87c9524580e28814de6f7dc194288 replica 3
>[kubeexec] DEBUG 2018/06/08 07:37:22 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script volume create vol_bcb87c9524580e28814de6f7dc194288 replica 3 10.70.47.76:/var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_b9d88edeed17a4ff0395a5b36b4cd7e0/brick 10.70.46.187:/var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_4481cc39f08c0bdcff489aab80756871/brick 10.70.46.122:/var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_4c6c93fd3801e871481fcd4fbf9b8ba8/brick
>Result: volume create: vol_bcb87c9524580e28814de6f7dc194288: success: please start the volume to access data
>[negroni] Started GET /queue/0a3a0f4fc1906a402387b85f96b23c96
>[negroni] Completed 200 OK in 108.708Âµs
>[negroni] Started GET /queue/0a3a0f4fc1906a402387b85f96b23c96
>[negroni] Completed 200 OK in 187.521Âµs
>[heketi] INFO 2018/06/08 07:37:24 Starting Node Health Status refresh
>[cmdexec] INFO 2018/06/08 07:37:24 Check Glusterd service status in node dhcp46-187.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 07:37:24 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:33 UTC; 1h 13min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48fed567_6ae4_11e8_bc4b_005056a5f18a.slice/docker-a506d9aaca4b440d184df40d01e90b2040811c8090bf4c19d892081307b28c0c.scope/system.slice/glusterd.service
>           ââ 433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ1228 /usr/sbin/glusterfsd -s 10.70.46.187 --volfile-id heketidbstorage.10.70.46.187.var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.187-var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.pid -S /var/run/gluster/c08898c2e3b0190c62df3a9d4a013c92.socket --brick-name /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.log --xlator-option *-posix.glusterd-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Unit glusterd.service entered failed state.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: glusterd.service failed.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:33 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 07:37:24 Periodic health check status: node 278bd6b4e16a8e62ef15aaae22e6abc1 up=true
>[cmdexec] INFO 2018/06/08 07:37:24 Check Glusterd service status in node dhcp46-122.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 07:37:24 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:42 UTC; 1h 13min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4900c968_6ae4_11e8_bc4b_005056a5f18a.slice/docker-438fb5fb16a82a3e701b28fa1f1ef8623dc37ca06374c604e5781a2b2ab2e06d.scope/system.slice/glusterd.service
>           ââ 433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ1106 /usr/sbin/glusterfsd -s 10.70.46.122 --volfile-id heketidbstorage.10.70.46.122.var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.122-var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.pid -S /var/run/gluster/9fa3ac3b12826912287adc88c26ac134.socket --brick-name /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_29e5b36d921cca1b0699871d9727c932/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.log --xlator-option *-posix.glusterd-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>
>Jun 08 06:23:40 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:42 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 07:37:24 Periodic health check status: node 70423cc0bcd044fe5ba8bbbd256a3e49 up=true
>[cmdexec] INFO 2018/06/08 07:37:24 Check Glusterd service status in node dhcp47-76.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 07:37:24 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script volume start vol_bcb87c9524580e28814de6f7dc194288
>Result: volume start: vol_bcb87c9524580e28814de6f7dc194288: success
>[heketi] INFO 2018/06/08 07:37:24 Create Volume succeeded
>[asynchttp] INFO 2018/06/08 07:37:24 asynchttp.go:292: Completed job 0a3a0f4fc1906a402387b85f96b23c96 in 3.889038852s
>[negroni] Started GET /queue/0a3a0f4fc1906a402387b85f96b23c96
>[negroni] Completed 303 See Other in 135.871Âµs
>[negroni] Started GET /volumes/bcb87c9524580e28814de6f7dc194288
>[negroni] Completed 200 OK in 10.076402ms
>[negroni] Started GET /clusters/0a73c60efdd4673113b668afea101e6d
>[negroni] Completed 200 OK in 220.398Âµs
>[negroni] Started GET /nodes/278bd6b4e16a8e62ef15aaae22e6abc1
>[negroni] Completed 200 OK in 1.039466ms
>[negroni] Started GET /nodes/70423cc0bcd044fe5ba8bbbd256a3e49
>[negroni] Completed 200 OK in 625.893Âµs
>[negroni] Started GET /nodes/d942fe6c0ee5691b7cc263968f97b650
>[negroni] Completed 200 OK in 649.202Âµs
>[kubeexec] DEBUG 2018/06/08 07:37:24 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 07:25:20 UTC; 12min ago
>  Process: 828 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 830 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5cb2299_6aec_11e8_bc4b_005056a5f18a.slice/docker-7f9223abc9a8641cea68f374f4039235515978ee2dab970ac6cbef182c535426.scope/system.slice/glusterd.service
>           ââ 830 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ1197 /usr/sbin/glusterfsd -s 10.70.47.76 --volfile-id heketidbstorage.10.70.47.76.var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.76-var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.pid -S /var/run/gluster/b6976f9ddc3abc90e1dfa91c35a7c611.socket --brick-name /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_54c06cdf52c6fe1e3a3f8a9d39cde33d/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.log --xlator-option *-posix.glusterd-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ3014 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/ec468453b11caed5259f5e1abcc65b44.socket --xlator-option *replicate*.node-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94
>           ââ3015 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/ec468453b11caed5259f5e1abcc65b44.socket --xlator-option *replicate*.node-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94
>
>Jun 08 07:25:17 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 07:25:20 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 07:37:24 Periodic health check status: node d942fe6c0ee5691b7cc263968f97b650 up=true
>[heketi] INFO 2018/06/08 07:37:24 Cleaned 0 nodes from health cache
>[negroni] Started DELETE /volumes/bcb87c9524580e28814de6f7dc194288
>[negroni] Completed 202 Accepted in 28.149131ms
>[asynchttp] INFO 2018/06/08 07:37:50 asynchttp.go:288: Started job f71c2513ed46d147319415ada119329a
>[heketi] INFO 2018/06/08 07:37:50 Started async operation: Delete Volume
>[negroni] Started GET /queue/f71c2513ed46d147319415ada119329a
>[negroni] Completed 200 OK in 135.257Âµs
>[kubeexec] DEBUG 2018/06/08 07:37:50 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: gluster --mode=script snapshot list vol_bcb87c9524580e28814de6f7dc194288 --xml
>Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
><cliOutput>
>  <opRet>0</opRet>
>  <opErrno>0</opErrno>
>  <opErrstr/>
>  <snapList>
>    <count>0</count>
>  </snapList>
></cliOutput>
>[negroni] Started GET /queue/f71c2513ed46d147319415ada119329a
>[negroni] Completed 200 OK in 204.058Âµs
>[negroni] Started GET /queue/f71c2513ed46d147319415ada119329a
>[negroni] Completed 200 OK in 119.564Âµs
>[kubeexec] DEBUG 2018/06/08 07:37:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: gluster --mode=script volume stop vol_bcb87c9524580e28814de6f7dc194288 force
>Result: volume stop: vol_bcb87c9524580e28814de6f7dc194288: success
>[kubeexec] DEBUG 2018/06/08 07:37:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: gluster --mode=script volume delete vol_bcb87c9524580e28814de6f7dc194288
>Result: volume delete: vol_bcb87c9524580e28814de6f7dc194288: success
>[heketi] INFO 2018/06/08 07:37:52 Deleting brick 4481cc39f08c0bdcff489aab80756871
>[heketi] INFO 2018/06/08 07:37:52 Deleting brick 4c6c93fd3801e871481fcd4fbf9b8ba8
>[heketi] INFO 2018/06/08 07:37:52 Deleting brick b9d88edeed17a4ff0395a5b36b4cd7e0
>[kubeexec] DEBUG 2018/06/08 07:37:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mount | grep -w /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_4c6c93fd3801e871481fcd4fbf9b8ba8 | cut -d" " -f1
>Result: /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_4c6c93fd3801e871481fcd4fbf9b8ba8
>[kubeexec] DEBUG 2018/06/08 07:37:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mount | grep -w /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_b9d88edeed17a4ff0395a5b36b4cd7e0 | cut -d" " -f1
>Result: /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_b9d88edeed17a4ff0395a5b36b4cd7e0
>[kubeexec] DEBUG 2018/06/08 07:37:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mount | grep -w /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_4481cc39f08c0bdcff489aab80756871 | cut -d" " -f1
>Result: /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_4481cc39f08c0bdcff489aab80756871
>[kubeexec] DEBUG 2018/06/08 07:37:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_b9d88edeed17a4ff0395a5b36b4cd7e0
>
>Result:   vg_96f1667f2f1ced2c5ef94772922be93b/tp_b9d88edeed17a4ff0395a5b36b4cd7e0
>[kubeexec] DEBUG 2018/06/08 07:37:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_4c6c93fd3801e871481fcd4fbf9b8ba8
>
>Result:   vg_3a4297677881963e3f80124971d50eea/tp_4c6c93fd3801e871481fcd4fbf9b8ba8
>[negroni] Started GET /queue/f71c2513ed46d147319415ada119329a
>[negroni] Completed 200 OK in 94.934Âµs
>[kubeexec] DEBUG 2018/06/08 07:37:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_4481cc39f08c0bdcff489aab80756871
>
>Result:   vg_d389f0278a774bd7443a09af960961d8/tp_4481cc39f08c0bdcff489aab80756871
>[kubeexec] DEBUG 2018/06/08 07:37:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: umount /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_b9d88edeed17a4ff0395a5b36b4cd7e0
>Result: 
>[negroni] Started GET /queue/f71c2513ed46d147319415ada119329a
>[negroni] Completed 200 OK in 102.101Âµs
>[kubeexec] DEBUG 2018/06/08 07:37:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: umount /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_4c6c93fd3801e871481fcd4fbf9b8ba8
>Result: 
>[kubeexec] DEBUG 2018/06/08 07:37:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: umount /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_4481cc39f08c0bdcff489aab80756871
>Result: 
>[negroni] Started GET /queue/f71c2513ed46d147319415ada119329a
>[negroni] Completed 200 OK in 121.371Âµs
>[kubeexec] DEBUG 2018/06/08 07:37:55 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: sed -i.save "/brick_b9d88edeed17a4ff0395a5b36b4cd7e0/d" /var/lib/heketi/fstab
>Result: 
>[kubeexec] DEBUG 2018/06/08 07:37:55 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: sed -i.save "/brick_4c6c93fd3801e871481fcd4fbf9b8ba8/d" /var/lib/heketi/fstab
>Result: 
>[negroni] Started GET /queue/f71c2513ed46d147319415ada119329a
>[negroni] Completed 200 OK in 121.211Âµs
>[kubeexec] DEBUG 2018/06/08 07:37:56 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: sed -i.save "/brick_4481cc39f08c0bdcff489aab80756871/d" /var/lib/heketi/fstab
>Result: 
>[kubeexec] DEBUG 2018/06/08 07:37:56 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvremove --autobackup=n -f /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_b9d88edeed17a4ff0395a5b36b4cd7e0
>
>Result:   Logical volume "brick_b9d88edeed17a4ff0395a5b36b4cd7e0" successfully removed
>[kubeexec] DEBUG 2018/06/08 07:37:56 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvremove --autobackup=n -f /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_4c6c93fd3801e871481fcd4fbf9b8ba8
>
>Result:   Logical volume "brick_4c6c93fd3801e871481fcd4fbf9b8ba8" successfully removed
>[negroni] Started GET /queue/f71c2513ed46d147319415ada119329a
>[negroni] Completed 200 OK in 125.686Âµs
>[kubeexec] DEBUG 2018/06/08 07:37:57 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvremove --autobackup=n -f /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_4481cc39f08c0bdcff489aab80756871
>
>Result:   Logical volume "brick_4481cc39f08c0bdcff489aab80756871" successfully removed
>[kubeexec] DEBUG 2018/06/08 07:37:57 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvs --noheadings --options=thin_count   vg_96f1667f2f1ced2c5ef94772922be93b/tp_b9d88edeed17a4ff0395a5b36b4cd7e0
>
>Result:        0
>[kubeexec] DEBUG 2018/06/08 07:37:57 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvs --noheadings --options=thin_count   vg_3a4297677881963e3f80124971d50eea/tp_4c6c93fd3801e871481fcd4fbf9b8ba8
>
>Result:        0
>[negroni] Started GET /queue/f71c2513ed46d147319415ada119329a
>[negroni] Completed 200 OK in 118.538Âµs
>[kubeexec] DEBUG 2018/06/08 07:37:58 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvs --noheadings --options=thin_count   vg_d389f0278a774bd7443a09af960961d8/tp_4481cc39f08c0bdcff489aab80756871
>
>Result:        0
>[kubeexec] DEBUG 2018/06/08 07:37:58 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvremove --autobackup=n -f   vg_96f1667f2f1ced2c5ef94772922be93b/tp_b9d88edeed17a4ff0395a5b36b4cd7e0
>
>Result:   Logical volume "tp_b9d88edeed17a4ff0395a5b36b4cd7e0" successfully removed
>[negroni] Started GET /queue/f71c2513ed46d147319415ada119329a
>[negroni] Completed 200 OK in 161.509Âµs
>[kubeexec] DEBUG 2018/06/08 07:37:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvremove --autobackup=n -f   vg_3a4297677881963e3f80124971d50eea/tp_4c6c93fd3801e871481fcd4fbf9b8ba8
>
>Result:   Logical volume "tp_4c6c93fd3801e871481fcd4fbf9b8ba8" successfully removed
>[kubeexec] DEBUG 2018/06/08 07:37:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvremove --autobackup=n -f   vg_d389f0278a774bd7443a09af960961d8/tp_4481cc39f08c0bdcff489aab80756871
>
>Result:   Logical volume "tp_4481cc39f08c0bdcff489aab80756871" successfully removed
>[kubeexec] DEBUG 2018/06/08 07:38:00 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: rmdir /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_b9d88edeed17a4ff0395a5b36b4cd7e0
>Result: 
>[negroni] Started GET /queue/f71c2513ed46d147319415ada119329a
>[negroni] Completed 200 OK in 118.425Âµs
>[kubeexec] DEBUG 2018/06/08 07:38:00 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: rmdir /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_4c6c93fd3801e871481fcd4fbf9b8ba8
>Result: 
>[kubeexec] DEBUG 2018/06/08 07:38:00 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: rmdir /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_4481cc39f08c0bdcff489aab80756871
>Result: 
>[heketi] INFO 2018/06/08 07:38:00 Delete Volume succeeded
>[asynchttp] INFO 2018/06/08 07:38:00 asynchttp.go:292: Completed job f71c2513ed46d147319415ada119329a in 10.566469437s
>[negroni] Started GET /queue/f71c2513ed46d147319415ada119329a
>[negroni] Completed 204 No Content in 162.342Âµs
>[heketi] INFO 2018/06/08 07:39:14 Starting Node Health Status refresh
>[cmdexec] INFO 2018/06/08 07:39:14 Check Glusterd service status in node dhcp46-187.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 07:39:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:33 UTC; 1h 15min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48fed567_6ae4_11e8_bc4b_005056a5f18a.slice/docker-a506d9aaca4b440d184df40d01e90b2040811c8090bf4c19d892081307b28c0c.scope/system.slice/glusterd.service
>           ââ 433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ1228 /usr/sbin/glusterfsd -s 10.70.46.187 --volfile-id heketidbstorage.10.70.46.187.var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.187-var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.pid -S /var/run/gluster/c08898c2e3b0190c62df3a9d4a013c92.socket --brick-name /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.log --xlator-option *-posix.glusterd-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ5639 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/8fb729a0bbd51b204491c603c84acaa4.socket --xlator-option *replicate*.node-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034
>
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Unit glusterd.service entered failed state.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: glusterd.service failed.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:33 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 07:39:14 Periodic health check status: node 278bd6b4e16a8e62ef15aaae22e6abc1 up=true
>[cmdexec] INFO 2018/06/08 07:39:14 Check Glusterd service status in node dhcp46-122.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 07:39:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:42 UTC; 1h 15min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4900c968_6ae4_11e8_bc4b_005056a5f18a.slice/docker-438fb5fb16a82a3e701b28fa1f1ef8623dc37ca06374c604e5781a2b2ab2e06d.scope/system.slice/glusterd.service
>           ââ 433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ1106 /usr/sbin/glusterfsd -s 10.70.46.122 --volfile-id heketidbstorage.10.70.46.122.var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.122-var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.pid -S /var/run/gluster/9fa3ac3b12826912287adc88c26ac134.socket --brick-name /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_29e5b36d921cca1b0699871d9727c932/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.log --xlator-option *-posix.glusterd-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ5506 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f8994cdb09351d83e70da08261d99c64.socket --xlator-option *replicate*.node-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0
>
>Jun 08 06:23:40 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:42 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 07:39:14 Periodic health check status: node 70423cc0bcd044fe5ba8bbbd256a3e49 up=true
>[cmdexec] INFO 2018/06/08 07:39:14 Check Glusterd service status in node dhcp47-76.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 07:39:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 07:25:20 UTC; 13min ago
>  Process: 828 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 830 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5cb2299_6aec_11e8_bc4b_005056a5f18a.slice/docker-7f9223abc9a8641cea68f374f4039235515978ee2dab970ac6cbef182c535426.scope/system.slice/glusterd.service
>           ââ 830 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ1197 /usr/sbin/glusterfsd -s 10.70.47.76 --volfile-id heketidbstorage.10.70.47.76.var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.76-var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.pid -S /var/run/gluster/b6976f9ddc3abc90e1dfa91c35a7c611.socket --brick-name /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_54c06cdf52c6fe1e3a3f8a9d39cde33d/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.log --xlator-option *-posix.glusterd-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ3079 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/ec468453b11caed5259f5e1abcc65b44.socket --xlator-option *replicate*.node-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94
>
>Jun 08 07:25:17 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 07:25:20 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 07:39:14 Periodic health check status: node d942fe6c0ee5691b7cc263968f97b650 up=true
>[heketi] INFO 2018/06/08 07:39:14 Cleaned 0 nodes from health cache
>[heketi] INFO 2018/06/08 07:41:14 Starting Node Health Status refresh
>[cmdexec] INFO 2018/06/08 07:41:14 Check Glusterd service status in node dhcp46-187.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 07:41:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:33 UTC; 1h 17min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48fed567_6ae4_11e8_bc4b_005056a5f18a.slice/docker-a506d9aaca4b440d184df40d01e90b2040811c8090bf4c19d892081307b28c0c.scope/system.slice/glusterd.service
>           ââ 433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ1228 /usr/sbin/glusterfsd -s 10.70.46.187 --volfile-id heketidbstorage.10.70.46.187.var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.187-var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.pid -S /var/run/gluster/c08898c2e3b0190c62df3a9d4a013c92.socket --brick-name /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.log --xlator-option *-posix.glusterd-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ5639 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/8fb729a0bbd51b204491c603c84acaa4.socket --xlator-option *replicate*.node-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034
>
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Unit glusterd.service entered failed state.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: glusterd.service failed.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:33 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 07:41:14 Periodic health check status: node 278bd6b4e16a8e62ef15aaae22e6abc1 up=true
>[cmdexec] INFO 2018/06/08 07:41:14 Check Glusterd service status in node dhcp46-122.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 07:41:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:42 UTC; 1h 17min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4900c968_6ae4_11e8_bc4b_005056a5f18a.slice/docker-438fb5fb16a82a3e701b28fa1f1ef8623dc37ca06374c604e5781a2b2ab2e06d.scope/system.slice/glusterd.service
>           ââ 433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ1106 /usr/sbin/glusterfsd -s 10.70.46.122 --volfile-id heketidbstorage.10.70.46.122.var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.122-var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.pid -S /var/run/gluster/9fa3ac3b12826912287adc88c26ac134.socket --brick-name /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_29e5b36d921cca1b0699871d9727c932/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.log --xlator-option *-posix.glusterd-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ5506 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f8994cdb09351d83e70da08261d99c64.socket --xlator-option *replicate*.node-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0
>
>Jun 08 06:23:40 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:42 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 07:41:14 Periodic health check status: node 70423cc0bcd044fe5ba8bbbd256a3e49 up=true
>[cmdexec] INFO 2018/06/08 07:41:14 Check Glusterd service status in node dhcp47-76.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 07:41:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 07:25:20 UTC; 15min ago
>  Process: 828 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 830 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5cb2299_6aec_11e8_bc4b_005056a5f18a.slice/docker-7f9223abc9a8641cea68f374f4039235515978ee2dab970ac6cbef182c535426.scope/system.slice/glusterd.service
>           ââ 830 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ1197 /usr/sbin/glusterfsd -s 10.70.47.76 --volfile-id heketidbstorage.10.70.47.76.var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.76-var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.pid -S /var/run/gluster/b6976f9ddc3abc90e1dfa91c35a7c611.socket --brick-name /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_54c06cdf52c6fe1e3a3f8a9d39cde33d/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.log --xlator-option *-posix.glusterd-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ3079 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/ec468453b11caed5259f5e1abcc65b44.socket --xlator-option *replicate*.node-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94
>
>Jun 08 07:25:17 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 07:25:20 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 07:41:14 Periodic health check status: node d942fe6c0ee5691b7cc263968f97b650 up=true
>[heketi] INFO 2018/06/08 07:41:14 Cleaned 0 nodes from health cache
>[negroni] Started POST /volumes
>[heketi] INFO 2018/06/08 07:41:33 Allocating brick set #0
>[negroni] Completed 202 Accepted in 13.669491ms
>[asynchttp] INFO 2018/06/08 07:41:33 asynchttp.go:288: Started job acac9d92f0d66231baa05a1d7ba787b0
>[heketi] INFO 2018/06/08 07:41:33 Started async operation: Create Volume
>[negroni] Started GET /queue/acac9d92f0d66231baa05a1d7ba787b0
>[negroni] Completed 200 OK in 161.499Âµs
>[heketi] INFO 2018/06/08 07:41:33 Creating brick e3b5c51759943388b0fc47d575ed5445
>[heketi] INFO 2018/06/08 07:41:33 Creating brick 8ae5b886d80baf6275776a0a1bf731ad
>[heketi] INFO 2018/06/08 07:41:33 Creating brick 08277a41c0781a41f0ec4d102be47ab8
>[kubeexec] DEBUG 2018/06/08 07:41:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir -p /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_e3b5c51759943388b0fc47d575ed5445
>Result: 
>[kubeexec] DEBUG 2018/06/08 07:41:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir -p /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_08277a41c0781a41f0ec4d102be47ab8
>Result: 
>[kubeexec] DEBUG 2018/06/08 07:41:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir -p /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_8ae5b886d80baf6275776a0a1bf731ad
>Result: 
>[kubeexec] DEBUG 2018/06/08 07:41:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvcreate --autobackup=n --poolmetadatasize 53248K --chunksize 256K --size 10485760K --thin vg_3a4297677881963e3f80124971d50eea/tp_8ae5b886d80baf6275776a0a1bf731ad --virtualsize 10485760K --name brick_8ae5b886d80baf6275776a0a1bf731ad
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_8ae5b886d80baf6275776a0a1bf731ad" created.
>[kubeexec] DEBUG 2018/06/08 07:41:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvcreate --autobackup=n --poolmetadatasize 53248K --chunksize 256K --size 10485760K --thin vg_b861af9cc67f5bf8d5a8f9e38f56b14e/tp_e3b5c51759943388b0fc47d575ed5445 --virtualsize 10485760K --name brick_e3b5c51759943388b0fc47d575ed5445
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_e3b5c51759943388b0fc47d575ed5445" created.
>[kubeexec] DEBUG 2018/06/08 07:41:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvcreate --autobackup=n --poolmetadatasize 53248K --chunksize 256K --size 10485760K --thin vg_9394bc70699b006c5460c9f654cf345f/tp_08277a41c0781a41f0ec4d102be47ab8 --virtualsize 10485760K --name brick_08277a41c0781a41f0ec4d102be47ab8
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_08277a41c0781a41f0ec4d102be47ab8" created.
>[kubeexec] DEBUG 2018/06/08 07:41:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_8ae5b886d80baf6275776a0a1bf731ad
>Result: meta-data=/dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_8ae5b886d80baf6275776a0a1bf731ad isize=512    agcount=16, agsize=163840 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=2621440, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 07:41:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_e3b5c51759943388b0fc47d575ed5445
>Result: meta-data=/dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_e3b5c51759943388b0fc47d575ed5445 isize=512    agcount=16, agsize=163840 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=2621440, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 07:41:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_08277a41c0781a41f0ec4d102be47ab8
>Result: meta-data=/dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_08277a41c0781a41f0ec4d102be47ab8 isize=512    agcount=16, agsize=163840 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=2621440, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 07:41:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: awk "BEGIN {print \"/dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_8ae5b886d80baf6275776a0a1bf731ad /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_8ae5b886d80baf6275776a0a1bf731ad xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 07:41:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: awk "BEGIN {print \"/dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_e3b5c51759943388b0fc47d575ed5445 /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_e3b5c51759943388b0fc47d575ed5445 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 07:41:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: awk "BEGIN {print \"/dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_08277a41c0781a41f0ec4d102be47ab8 /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_08277a41c0781a41f0ec4d102be47ab8 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 07:41:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_8ae5b886d80baf6275776a0a1bf731ad /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_8ae5b886d80baf6275776a0a1bf731ad
>Result: 
>[kubeexec] DEBUG 2018/06/08 07:41:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_e3b5c51759943388b0fc47d575ed5445 /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_e3b5c51759943388b0fc47d575ed5445
>Result: 
>[kubeexec] DEBUG 2018/06/08 07:41:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_08277a41c0781a41f0ec4d102be47ab8 /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_08277a41c0781a41f0ec4d102be47ab8
>Result: 
>[negroni] Started GET /queue/acac9d92f0d66231baa05a1d7ba787b0
>[negroni] Completed 200 OK in 114.777Âµs
>[kubeexec] DEBUG 2018/06/08 07:41:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_8ae5b886d80baf6275776a0a1bf731ad/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 07:41:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_08277a41c0781a41f0ec4d102be47ab8/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 07:41:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_e3b5c51759943388b0fc47d575ed5445/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 07:41:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: chown :2001 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_8ae5b886d80baf6275776a0a1bf731ad/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 07:41:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: chown :2001 /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_08277a41c0781a41f0ec4d102be47ab8/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 07:41:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: chown :2001 /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_e3b5c51759943388b0fc47d575ed5445/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 07:41:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: chmod 2775 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_8ae5b886d80baf6275776a0a1bf731ad/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 07:41:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: chmod 2775 /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_08277a41c0781a41f0ec4d102be47ab8/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 07:41:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: chmod 2775 /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_e3b5c51759943388b0fc47d575ed5445/brick
>Result: 
>[cmdexec] INFO 2018/06/08 07:41:34 Creating volume cns-vol_glusterfs_mongodb5_5d4f9043-6aef-11e8-ab19-005056a5f18a replica 3
>[kubeexec] DEBUG 2018/06/08 07:41:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: gluster --mode=script volume create cns-vol_glusterfs_mongodb5_5d4f9043-6aef-11e8-ab19-005056a5f18a replica 3 10.70.46.187:/var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_e3b5c51759943388b0fc47d575ed5445/brick 10.70.46.122:/var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_8ae5b886d80baf6275776a0a1bf731ad/brick 10.70.47.76:/var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_08277a41c0781a41f0ec4d102be47ab8/brick
>Result: volume create: cns-vol_glusterfs_mongodb5_5d4f9043-6aef-11e8-ab19-005056a5f18a: success: please start the volume to access data
>[negroni] Started GET /queue/acac9d92f0d66231baa05a1d7ba787b0
>[negroni] Completed 200 OK in 137.655Âµs
>[negroni] Started GET /queue/acac9d92f0d66231baa05a1d7ba787b0
>[negroni] Completed 200 OK in 127.992Âµs
>[negroni] Started GET /queue/acac9d92f0d66231baa05a1d7ba787b0
>[negroni] Completed 200 OK in 133.277Âµs
>[negroni] Started GET /queue/acac9d92f0d66231baa05a1d7ba787b0
>[negroni] Completed 200 OK in 149.179Âµs
>[kubeexec] DEBUG 2018/06/08 07:41:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: gluster --mode=script volume start cns-vol_glusterfs_mongodb5_5d4f9043-6aef-11e8-ab19-005056a5f18a
>Result: volume start: cns-vol_glusterfs_mongodb5_5d4f9043-6aef-11e8-ab19-005056a5f18a: success
>[heketi] INFO 2018/06/08 07:41:38 Create Volume succeeded
>[asynchttp] INFO 2018/06/08 07:41:38 asynchttp.go:292: Completed job acac9d92f0d66231baa05a1d7ba787b0 in 5.804225171s
>[negroni] Started GET /queue/acac9d92f0d66231baa05a1d7ba787b0
>[negroni] Completed 303 See Other in 276.781Âµs
>[negroni] Started GET /volumes/24485cd4cac31cedd877a92a75f2397f
>[negroni] Completed 200 OK in 2.268844ms
>[negroni] Started GET /clusters/0a73c60efdd4673113b668afea101e6d
>[negroni] Completed 200 OK in 332.308Âµs
>[negroni] Started GET /nodes/278bd6b4e16a8e62ef15aaae22e6abc1
>[negroni] Completed 200 OK in 973.725Âµs
>[negroni] Started GET /nodes/70423cc0bcd044fe5ba8bbbd256a3e49
>[negroni] Completed 200 OK in 723.558Âµs
>[negroni] Started GET /nodes/d942fe6c0ee5691b7cc263968f97b650
>[negroni] Completed 200 OK in 593.338Âµs
>[negroni] Started DELETE /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 202 Accepted in 10.044724ms
>[asynchttp] INFO 2018/06/08 07:42:20 asynchttp.go:288: Started job c62f358eeb3193ad12ba712fd51ac30c
>[heketi] INFO 2018/06/08 07:42:20 Started async operation: Delete Volume
>[negroni] Started GET /queue/c62f358eeb3193ad12ba712fd51ac30c
>[negroni] Completed 200 OK in 151.979Âµs
>[kubeexec] DEBUG 2018/06/08 07:42:20 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script snapshot list vol_9c7d5f0f473cce6914804135f0b8ddcd --xml
>Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
><cliOutput>
>  <opRet>-1</opRet>
>  <opErrno>30806</opErrno>
>  <opErrstr>Volume (vol_9c7d5f0f473cce6914804135f0b8ddcd) does not exist</opErrstr>
></cliOutput>
>[kubeexec] ERROR 2018/06/08 07:42:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume stop vol_9c7d5f0f473cce6914804135f0b8ddcd force] on glusterfs-storage-gxp7c: Err[command terminated with exit code 1]: Stdout []: Stderr [volume stop: vol_9c7d5f0f473cce6914804135f0b8ddcd: failed: Volume vol_9c7d5f0f473cce6914804135f0b8ddcd does not exist
>]
>[cmdexec] ERROR 2018/06/08 07:42:21 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:144: Unable to stop volume vol_9c7d5f0f473cce6914804135f0b8ddcd: Unable to execute command on glusterfs-storage-gxp7c: volume stop: vol_9c7d5f0f473cce6914804135f0b8ddcd: failed: Volume vol_9c7d5f0f473cce6914804135f0b8ddcd does not exist
>[kubeexec] ERROR 2018/06/08 07:42:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume delete vol_9c7d5f0f473cce6914804135f0b8ddcd] on glusterfs-storage-gxp7c: Err[command terminated with exit code 1]: Stdout []: Stderr [volume delete: vol_9c7d5f0f473cce6914804135f0b8ddcd: failed: Volume vol_9c7d5f0f473cce6914804135f0b8ddcd does not exist
>]
>[cmdexec] ERROR 2018/06/08 07:42:21 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:153: Unable to delete volume vol_9c7d5f0f473cce6914804135f0b8ddcd: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c7d5f0f473cce6914804135f0b8ddcd: failed: Volume vol_9c7d5f0f473cce6914804135f0b8ddcd does not exist
>[heketi] ERROR 2018/06/08 07:42:21 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:525: Unable to delete volume: Unable to delete volume vol_9c7d5f0f473cce6914804135f0b8ddcd: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c7d5f0f473cce6914804135f0b8ddcd: failed: Volume vol_9c7d5f0f473cce6914804135f0b8ddcd does not exist
>[heketi] ERROR 2018/06/08 07:42:21 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:429: Error executing delete volume: Unable to delete volume vol_9c7d5f0f473cce6914804135f0b8ddcd: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c7d5f0f473cce6914804135f0b8ddcd: failed: Volume vol_9c7d5f0f473cce6914804135f0b8ddcd does not exist
>[heketi] ERROR 2018/06/08 07:42:21 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:1185: Delete Volume Failed: Unable to delete volume vol_9c7d5f0f473cce6914804135f0b8ddcd: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c7d5f0f473cce6914804135f0b8ddcd: failed: Volume vol_9c7d5f0f473cce6914804135f0b8ddcd does not exist
>[asynchttp] INFO 2018/06/08 07:42:21 asynchttp.go:292: Completed job c62f358eeb3193ad12ba712fd51ac30c in 721.195243ms
>[negroni] Started GET /queue/c62f358eeb3193ad12ba712fd51ac30c
>[negroni] Completed 500 Internal Server Error in 138.538Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 401 Unauthorized in 185.442Âµs
>[heketi] INFO 2018/06/08 07:43:14 Starting Node Health Status refresh
>[cmdexec] INFO 2018/06/08 07:43:14 Check Glusterd service status in node dhcp46-187.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 07:43:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:33 UTC; 1h 19min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48fed567_6ae4_11e8_bc4b_005056a5f18a.slice/docker-a506d9aaca4b440d184df40d01e90b2040811c8090bf4c19d892081307b28c0c.scope/system.slice/glusterd.service
>           ââ 433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ1228 /usr/sbin/glusterfsd -s 10.70.46.187 --volfile-id heketidbstorage.10.70.46.187.var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.187-var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.pid -S /var/run/gluster/c08898c2e3b0190c62df3a9d4a013c92.socket --brick-name /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.log --xlator-option *-posix.glusterd-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ5986 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/8fb729a0bbd51b204491c603c84acaa4.socket --xlator-option *replicate*.node-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034
>
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Unit glusterd.service entered failed state.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: glusterd.service failed.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:33 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 07:43:14 Periodic health check status: node 278bd6b4e16a8e62ef15aaae22e6abc1 up=true
>[cmdexec] INFO 2018/06/08 07:43:14 Check Glusterd service status in node dhcp46-122.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 07:43:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:42 UTC; 1h 19min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4900c968_6ae4_11e8_bc4b_005056a5f18a.slice/docker-438fb5fb16a82a3e701b28fa1f1ef8623dc37ca06374c604e5781a2b2ab2e06d.scope/system.slice/glusterd.service
>           ââ 433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ1106 /usr/sbin/glusterfsd -s 10.70.46.122 --volfile-id heketidbstorage.10.70.46.122.var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.122-var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.pid -S /var/run/gluster/9fa3ac3b12826912287adc88c26ac134.socket --brick-name /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_29e5b36d921cca1b0699871d9727c932/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.log --xlator-option *-posix.glusterd-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ5802 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f8994cdb09351d83e70da08261d99c64.socket --xlator-option *replicate*.node-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0
>
>Jun 08 06:23:40 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:42 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 07:43:14 Periodic health check status: node 70423cc0bcd044fe5ba8bbbd256a3e49 up=true
>[cmdexec] INFO 2018/06/08 07:43:14 Check Glusterd service status in node dhcp47-76.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 07:43:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 07:25:20 UTC; 17min ago
>  Process: 828 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 830 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5cb2299_6aec_11e8_bc4b_005056a5f18a.slice/docker-7f9223abc9a8641cea68f374f4039235515978ee2dab970ac6cbef182c535426.scope/system.slice/glusterd.service
>           ââ 830 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ1197 /usr/sbin/glusterfsd -s 10.70.47.76 --volfile-id heketidbstorage.10.70.47.76.var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.76-var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.pid -S /var/run/gluster/b6976f9ddc3abc90e1dfa91c35a7c611.socket --brick-name /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_54c06cdf52c6fe1e3a3f8a9d39cde33d/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.log --xlator-option *-posix.glusterd-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ3374 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/ec468453b11caed5259f5e1abcc65b44.socket --xlator-option *replicate*.node-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94
>
>Jun 08 07:25:17 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 07:25:20 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 07:43:14 Periodic health check status: node d942fe6c0ee5691b7cc263968f97b650 up=true
>[heketi] INFO 2018/06/08 07:43:14 Cleaned 0 nodes from health cache
>[negroni] Started POST /volumes
>[heketi] INFO 2018/06/08 07:44:39 Allocating brick set #0
>[negroni] Completed 202 Accepted in 18.037248ms
>[asynchttp] INFO 2018/06/08 07:44:39 asynchttp.go:288: Started job 79b4dee22f486972d3aef2af0be6cc0b
>[heketi] INFO 2018/06/08 07:44:39 Started async operation: Create Volume
>[negroni] Started GET /queue/79b4dee22f486972d3aef2af0be6cc0b
>[negroni] Completed 200 OK in 162.93Âµs
>[heketi] INFO 2018/06/08 07:44:39 Creating brick 8f853bf979222c0f3eec071f7605f2c2
>[heketi] INFO 2018/06/08 07:44:39 Creating brick 6fe37e54fbf6c9bb1e66e24c971d7a76
>[heketi] INFO 2018/06/08 07:44:39 Creating brick fa4c6b50ba3c59db3faff8234b61e970
>[kubeexec] DEBUG 2018/06/08 07:44:39 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir -p /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_8f853bf979222c0f3eec071f7605f2c2
>Result: 
>[kubeexec] DEBUG 2018/06/08 07:44:39 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir -p /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_fa4c6b50ba3c59db3faff8234b61e970
>Result: 
>[kubeexec] DEBUG 2018/06/08 07:44:40 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir -p /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_6fe37e54fbf6c9bb1e66e24c971d7a76
>Result: 
>[kubeexec] DEBUG 2018/06/08 07:44:40 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvcreate --autobackup=n --poolmetadatasize 53248K --chunksize 256K --size 10485760K --thin vg_3a4297677881963e3f80124971d50eea/tp_fa4c6b50ba3c59db3faff8234b61e970 --virtualsize 10485760K --name brick_fa4c6b50ba3c59db3faff8234b61e970
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_fa4c6b50ba3c59db3faff8234b61e970" created.
>[kubeexec] DEBUG 2018/06/08 07:44:40 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvcreate --autobackup=n --poolmetadatasize 53248K --chunksize 256K --size 10485760K --thin vg_d389f0278a774bd7443a09af960961d8/tp_8f853bf979222c0f3eec071f7605f2c2 --virtualsize 10485760K --name brick_8f853bf979222c0f3eec071f7605f2c2
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_8f853bf979222c0f3eec071f7605f2c2" created.
>[kubeexec] DEBUG 2018/06/08 07:44:40 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvcreate --autobackup=n --poolmetadatasize 53248K --chunksize 256K --size 10485760K --thin vg_96f1667f2f1ced2c5ef94772922be93b/tp_6fe37e54fbf6c9bb1e66e24c971d7a76 --virtualsize 10485760K --name brick_6fe37e54fbf6c9bb1e66e24c971d7a76
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_6fe37e54fbf6c9bb1e66e24c971d7a76" created.
>[kubeexec] DEBUG 2018/06/08 07:44:40 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_8f853bf979222c0f3eec071f7605f2c2
>Result: meta-data=/dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_8f853bf979222c0f3eec071f7605f2c2 isize=512    agcount=16, agsize=163840 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=2621440, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 07:44:40 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_6fe37e54fbf6c9bb1e66e24c971d7a76
>Result: meta-data=/dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_6fe37e54fbf6c9bb1e66e24c971d7a76 isize=512    agcount=16, agsize=163840 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=2621440, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 07:44:40 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: awk "BEGIN {print \"/dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_8f853bf979222c0f3eec071f7605f2c2 /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_8f853bf979222c0f3eec071f7605f2c2 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 07:44:40 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: awk "BEGIN {print \"/dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_6fe37e54fbf6c9bb1e66e24c971d7a76 /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_6fe37e54fbf6c9bb1e66e24c971d7a76 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 07:44:40 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_8f853bf979222c0f3eec071f7605f2c2 /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_8f853bf979222c0f3eec071f7605f2c2
>Result: 
>[kubeexec] DEBUG 2018/06/08 07:44:40 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_fa4c6b50ba3c59db3faff8234b61e970
>Result: meta-data=/dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_fa4c6b50ba3c59db3faff8234b61e970 isize=512    agcount=16, agsize=163840 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=2621440, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 07:44:40 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_6fe37e54fbf6c9bb1e66e24c971d7a76 /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_6fe37e54fbf6c9bb1e66e24c971d7a76
>Result: 
>[negroni] Started GET /queue/79b4dee22f486972d3aef2af0be6cc0b
>[negroni] Completed 200 OK in 101.912Âµs
>[kubeexec] DEBUG 2018/06/08 07:44:40 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_8f853bf979222c0f3eec071f7605f2c2/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 07:44:40 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: awk "BEGIN {print \"/dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_fa4c6b50ba3c59db3faff8234b61e970 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_fa4c6b50ba3c59db3faff8234b61e970 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 07:44:40 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_6fe37e54fbf6c9bb1e66e24c971d7a76/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 07:44:40 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: chown :2001 /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_8f853bf979222c0f3eec071f7605f2c2/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 07:44:40 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: chown :2001 /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_6fe37e54fbf6c9bb1e66e24c971d7a76/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 07:44:41 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_fa4c6b50ba3c59db3faff8234b61e970 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_fa4c6b50ba3c59db3faff8234b61e970
>Result: 
>[kubeexec] DEBUG 2018/06/08 07:44:41 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: chmod 2775 /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_8f853bf979222c0f3eec071f7605f2c2/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 07:44:41 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: chmod 2775 /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_6fe37e54fbf6c9bb1e66e24c971d7a76/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 07:44:41 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_fa4c6b50ba3c59db3faff8234b61e970/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 07:44:41 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: chown :2001 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_fa4c6b50ba3c59db3faff8234b61e970/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 07:44:41 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: chmod 2775 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_fa4c6b50ba3c59db3faff8234b61e970/brick
>Result: 
>[cmdexec] INFO 2018/06/08 07:44:41 Creating volume cns-vol_glusterfs_mongodb5_cca32e68-6aef-11e8-ab19-005056a5f18a replica 3
>[kubeexec] DEBUG 2018/06/08 07:44:41 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script volume create cns-vol_glusterfs_mongodb5_cca32e68-6aef-11e8-ab19-005056a5f18a replica 3 10.70.47.76:/var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_6fe37e54fbf6c9bb1e66e24c971d7a76/brick 10.70.46.187:/var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_8f853bf979222c0f3eec071f7605f2c2/brick 10.70.46.122:/var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_fa4c6b50ba3c59db3faff8234b61e970/brick
>Result: volume create: cns-vol_glusterfs_mongodb5_cca32e68-6aef-11e8-ab19-005056a5f18a: success: please start the volume to access data
>[negroni] Started GET /queue/79b4dee22f486972d3aef2af0be6cc0b
>[negroni] Completed 200 OK in 118.906Âµs
>[negroni] Started GET /queue/79b4dee22f486972d3aef2af0be6cc0b
>[negroni] Completed 200 OK in 114.343Âµs
>[negroni] Started GET /queue/79b4dee22f486972d3aef2af0be6cc0b
>[negroni] Completed 200 OK in 121.014Âµs
>[negroni] Started GET /queue/79b4dee22f486972d3aef2af0be6cc0b
>[negroni] Completed 200 OK in 187.376Âµs
>[kubeexec] DEBUG 2018/06/08 07:44:44 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script volume start cns-vol_glusterfs_mongodb5_cca32e68-6aef-11e8-ab19-005056a5f18a
>Result: volume start: cns-vol_glusterfs_mongodb5_cca32e68-6aef-11e8-ab19-005056a5f18a: success
>[heketi] INFO 2018/06/08 07:44:44 Create Volume succeeded
>[asynchttp] INFO 2018/06/08 07:44:44 asynchttp.go:292: Completed job 79b4dee22f486972d3aef2af0be6cc0b in 5.037729966s
>[negroni] Started GET /queue/79b4dee22f486972d3aef2af0be6cc0b
>[negroni] Completed 303 See Other in 139.004Âµs
>[negroni] Started GET /volumes/c0229543f8974d87bd07caae4493ca58
>[negroni] Completed 200 OK in 2.265835ms
>[negroni] Started GET /clusters/0a73c60efdd4673113b668afea101e6d
>[negroni] Completed 200 OK in 174.272Âµs
>[negroni] Started GET /nodes/278bd6b4e16a8e62ef15aaae22e6abc1
>[negroni] Completed 200 OK in 1.048723ms
>[negroni] Started GET /nodes/70423cc0bcd044fe5ba8bbbd256a3e49
>[negroni] Completed 200 OK in 1.033049ms
>[negroni] Started GET /nodes/d942fe6c0ee5691b7cc263968f97b650
>[negroni] Completed 200 OK in 829.401Âµs
>[negroni] Started DELETE /volumes/24485cd4cac31cedd877a92a75f2397f
>[negroni] Completed 202 Accepted in 10.474137ms
>[asynchttp] INFO 2018/06/08 07:44:50 asynchttp.go:288: Started job ef78edaa984b016aa2ccbe1ec46ac389
>[heketi] INFO 2018/06/08 07:44:50 Started async operation: Delete Volume
>[negroni] Started GET /queue/ef78edaa984b016aa2ccbe1ec46ac389
>[negroni] Completed 200 OK in 207.625Âµs
>[kubeexec] DEBUG 2018/06/08 07:44:50 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script snapshot list cns-vol_glusterfs_mongodb5_5d4f9043-6aef-11e8-ab19-005056a5f18a --xml
>Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
><cliOutput>
>  <opRet>0</opRet>
>  <opErrno>0</opErrno>
>  <opErrstr/>
>  <snapList>
>    <count>0</count>
>  </snapList>
></cliOutput>
>[negroni] Started GET /queue/ef78edaa984b016aa2ccbe1ec46ac389
>[negroni] Completed 200 OK in 124.076Âµs
>[negroni] Started GET /queue/ef78edaa984b016aa2ccbe1ec46ac389
>[negroni] Completed 200 OK in 141.987Âµs
>[kubeexec] DEBUG 2018/06/08 07:44:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script volume stop cns-vol_glusterfs_mongodb5_5d4f9043-6aef-11e8-ab19-005056a5f18a force
>Result: volume stop: cns-vol_glusterfs_mongodb5_5d4f9043-6aef-11e8-ab19-005056a5f18a: success
>[kubeexec] DEBUG 2018/06/08 07:44:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script volume delete cns-vol_glusterfs_mongodb5_5d4f9043-6aef-11e8-ab19-005056a5f18a
>Result: volume delete: cns-vol_glusterfs_mongodb5_5d4f9043-6aef-11e8-ab19-005056a5f18a: success
>[heketi] INFO 2018/06/08 07:44:53 Deleting brick 8ae5b886d80baf6275776a0a1bf731ad
>[heketi] INFO 2018/06/08 07:44:53 Deleting brick e3b5c51759943388b0fc47d575ed5445
>[heketi] INFO 2018/06/08 07:44:53 Deleting brick 08277a41c0781a41f0ec4d102be47ab8
>[negroni] Started GET /queue/ef78edaa984b016aa2ccbe1ec46ac389
>[negroni] Completed 200 OK in 127.611Âµs
>[kubeexec] DEBUG 2018/06/08 07:44:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mount | grep -w /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_e3b5c51759943388b0fc47d575ed5445 | cut -d" " -f1
>Result: /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_e3b5c51759943388b0fc47d575ed5445
>[kubeexec] DEBUG 2018/06/08 07:44:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mount | grep -w /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_08277a41c0781a41f0ec4d102be47ab8 | cut -d" " -f1
>Result: /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_08277a41c0781a41f0ec4d102be47ab8
>[kubeexec] DEBUG 2018/06/08 07:44:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mount | grep -w /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_8ae5b886d80baf6275776a0a1bf731ad | cut -d" " -f1
>Result: /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_8ae5b886d80baf6275776a0a1bf731ad
>[kubeexec] DEBUG 2018/06/08 07:44:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_e3b5c51759943388b0fc47d575ed5445
>
>Result:   vg_b861af9cc67f5bf8d5a8f9e38f56b14e/tp_e3b5c51759943388b0fc47d575ed5445
>[kubeexec] DEBUG 2018/06/08 07:44:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_08277a41c0781a41f0ec4d102be47ab8
>
>Result:   vg_9394bc70699b006c5460c9f654cf345f/tp_08277a41c0781a41f0ec4d102be47ab8
>[kubeexec] DEBUG 2018/06/08 07:44:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_8ae5b886d80baf6275776a0a1bf731ad
>
>Result:   vg_3a4297677881963e3f80124971d50eea/tp_8ae5b886d80baf6275776a0a1bf731ad
>[negroni] Started GET /queue/ef78edaa984b016aa2ccbe1ec46ac389
>[negroni] Completed 200 OK in 104.172Âµs
>[kubeexec] DEBUG 2018/06/08 07:44:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: umount /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_e3b5c51759943388b0fc47d575ed5445
>Result: 
>[kubeexec] DEBUG 2018/06/08 07:44:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: umount /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_08277a41c0781a41f0ec4d102be47ab8
>Result: 
>[negroni] Started GET /queue/ef78edaa984b016aa2ccbe1ec46ac389
>[negroni] Completed 200 OK in 134.163Âµs
>[kubeexec] DEBUG 2018/06/08 07:44:55 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: umount /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_8ae5b886d80baf6275776a0a1bf731ad
>Result: 
>[kubeexec] DEBUG 2018/06/08 07:44:55 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: sed -i.save "/brick_e3b5c51759943388b0fc47d575ed5445/d" /var/lib/heketi/fstab
>Result: 
>[kubeexec] DEBUG 2018/06/08 07:44:56 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: sed -i.save "/brick_08277a41c0781a41f0ec4d102be47ab8/d" /var/lib/heketi/fstab
>Result: 
>[negroni] Started GET /queue/ef78edaa984b016aa2ccbe1ec46ac389
>[negroni] Completed 200 OK in 116.018Âµs
>[kubeexec] DEBUG 2018/06/08 07:44:56 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: sed -i.save "/brick_8ae5b886d80baf6275776a0a1bf731ad/d" /var/lib/heketi/fstab
>Result: 
>[kubeexec] DEBUG 2018/06/08 07:44:57 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvremove --autobackup=n -f /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_e3b5c51759943388b0fc47d575ed5445
>
>Result:   Logical volume "brick_e3b5c51759943388b0fc47d575ed5445" successfully removed
>[kubeexec] DEBUG 2018/06/08 07:44:57 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvremove --autobackup=n -f /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_08277a41c0781a41f0ec4d102be47ab8
>
>Result:   Logical volume "brick_08277a41c0781a41f0ec4d102be47ab8" successfully removed
>[negroni] Started GET /queue/ef78edaa984b016aa2ccbe1ec46ac389
>[negroni] Completed 200 OK in 101.951Âµs
>[kubeexec] DEBUG 2018/06/08 07:44:58 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvremove --autobackup=n -f /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_8ae5b886d80baf6275776a0a1bf731ad
>
>Result:   Logical volume "brick_8ae5b886d80baf6275776a0a1bf731ad" successfully removed
>[kubeexec] DEBUG 2018/06/08 07:44:58 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvs --noheadings --options=thin_count   vg_b861af9cc67f5bf8d5a8f9e38f56b14e/tp_e3b5c51759943388b0fc47d575ed5445
>
>Result:        0
>[kubeexec] DEBUG 2018/06/08 07:44:58 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvs --noheadings --options=thin_count   vg_9394bc70699b006c5460c9f654cf345f/tp_08277a41c0781a41f0ec4d102be47ab8
>
>Result:        0
>[negroni] Started GET /queue/ef78edaa984b016aa2ccbe1ec46ac389
>[negroni] Completed 200 OK in 93.587Âµs
>[kubeexec] DEBUG 2018/06/08 07:44:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvs --noheadings --options=thin_count   vg_3a4297677881963e3f80124971d50eea/tp_8ae5b886d80baf6275776a0a1bf731ad
>
>Result:        0
>[kubeexec] DEBUG 2018/06/08 07:44:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvremove --autobackup=n -f   vg_b861af9cc67f5bf8d5a8f9e38f56b14e/tp_e3b5c51759943388b0fc47d575ed5445
>
>Result:   Logical volume "tp_e3b5c51759943388b0fc47d575ed5445" successfully removed
>[negroni] Started GET /queue/ef78edaa984b016aa2ccbe1ec46ac389
>[negroni] Completed 200 OK in 115.438Âµs
>[kubeexec] DEBUG 2018/06/08 07:44:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvremove --autobackup=n -f   vg_9394bc70699b006c5460c9f654cf345f/tp_08277a41c0781a41f0ec4d102be47ab8
>
>Result:   Logical volume "tp_08277a41c0781a41f0ec4d102be47ab8" successfully removed
>[kubeexec] DEBUG 2018/06/08 07:45:00 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvremove --autobackup=n -f   vg_3a4297677881963e3f80124971d50eea/tp_8ae5b886d80baf6275776a0a1bf731ad
>
>Result:   Logical volume "tp_8ae5b886d80baf6275776a0a1bf731ad" successfully removed
>[negroni] Started GET /queue/ef78edaa984b016aa2ccbe1ec46ac389
>[negroni] Completed 200 OK in 113.359Âµs
>[kubeexec] DEBUG 2018/06/08 07:45:00 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: rmdir /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_e3b5c51759943388b0fc47d575ed5445
>Result: 
>[kubeexec] DEBUG 2018/06/08 07:45:00 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: rmdir /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_08277a41c0781a41f0ec4d102be47ab8
>Result: 
>[negroni] Started GET /volumes
>[negroni] Completed 401 Unauthorized in 111.511Âµs
>[kubeexec] DEBUG 2018/06/08 07:45:01 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: rmdir /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_8ae5b886d80baf6275776a0a1bf731ad
>Result: 
>[heketi] INFO 2018/06/08 07:45:01 Delete Volume succeeded
>[asynchttp] INFO 2018/06/08 07:45:01 asynchttp.go:292: Completed job ef78edaa984b016aa2ccbe1ec46ac389 in 10.670538755s
>[negroni] Started GET /queue/ef78edaa984b016aa2ccbe1ec46ac389
>[negroni] Completed 204 No Content in 111.504Âµs
>[negroni] Started DELETE /volumes/24485cd4cac31cedd877a92a75f2397f
>[negroni] Completed 404 Not Found in 1.871355ms
>[negroni] Started GET /volumes
>[negroni] Completed 401 Unauthorized in 189.343Âµs
>[heketi] INFO 2018/06/08 07:45:14 Starting Node Health Status refresh
>[cmdexec] INFO 2018/06/08 07:45:14 Check Glusterd service status in node dhcp46-187.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 07:45:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:33 UTC; 1h 21min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48fed567_6ae4_11e8_bc4b_005056a5f18a.slice/docker-a506d9aaca4b440d184df40d01e90b2040811c8090bf4c19d892081307b28c0c.scope/system.slice/glusterd.service
>           ââ 433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ1228 /usr/sbin/glusterfsd -s 10.70.46.187 --volfile-id heketidbstorage.10.70.46.187.var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.187-var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.pid -S /var/run/gluster/c08898c2e3b0190c62df3a9d4a013c92.socket --brick-name /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.log --xlator-option *-posix.glusterd-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ6262 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/8fb729a0bbd51b204491c603c84acaa4.socket --xlator-option *replicate*.node-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034
>
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Unit glusterd.service entered failed state.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: glusterd.service failed.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:33 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 07:45:14 Periodic health check status: node 278bd6b4e16a8e62ef15aaae22e6abc1 up=true
>[cmdexec] INFO 2018/06/08 07:45:14 Check Glusterd service status in node dhcp46-122.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 07:45:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:42 UTC; 1h 21min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4900c968_6ae4_11e8_bc4b_005056a5f18a.slice/docker-438fb5fb16a82a3e701b28fa1f1ef8623dc37ca06374c604e5781a2b2ab2e06d.scope/system.slice/glusterd.service
>           ââ 433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ1106 /usr/sbin/glusterfsd -s 10.70.46.122 --volfile-id heketidbstorage.10.70.46.122.var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.122-var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.pid -S /var/run/gluster/9fa3ac3b12826912287adc88c26ac134.socket --brick-name /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_29e5b36d921cca1b0699871d9727c932/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.log --xlator-option *-posix.glusterd-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ6078 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f8994cdb09351d83e70da08261d99c64.socket --xlator-option *replicate*.node-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0
>
>Jun 08 06:23:40 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:42 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 07:45:14 Periodic health check status: node 70423cc0bcd044fe5ba8bbbd256a3e49 up=true
>[cmdexec] INFO 2018/06/08 07:45:14 Check Glusterd service status in node dhcp47-76.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 07:45:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 07:25:20 UTC; 19min ago
>  Process: 828 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 830 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5cb2299_6aec_11e8_bc4b_005056a5f18a.slice/docker-7f9223abc9a8641cea68f374f4039235515978ee2dab970ac6cbef182c535426.scope/system.slice/glusterd.service
>           ââ 830 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ1197 /usr/sbin/glusterfsd -s 10.70.47.76 --volfile-id heketidbstorage.10.70.47.76.var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.76-var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.pid -S /var/run/gluster/b6976f9ddc3abc90e1dfa91c35a7c611.socket --brick-name /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_54c06cdf52c6fe1e3a3f8a9d39cde33d/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.log --xlator-option *-posix.glusterd-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ3763 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/ec468453b11caed5259f5e1abcc65b44.socket --xlator-option *replicate*.node-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94
>
>Jun 08 07:25:17 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 07:25:20 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 07:45:14 Periodic health check status: node d942fe6c0ee5691b7cc263968f97b650 up=true
>[heketi] INFO 2018/06/08 07:45:14 Cleaned 0 nodes from health cache
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 1.249431ms
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 1.494658ms
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 940.927Âµs
>[negroni] Started GET /volumes/c0229543f8974d87bd07caae4493ca58
>[negroni] Completed 200 OK in 1.004026ms
>[negroni] Started GET /volumes/e6860021031e7cf362c2b3824d6c351f
>[negroni] Completed 200 OK in 586.985Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 197.659Âµs
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 702.563Âµs
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 550.77Âµs
>[negroni] Started GET /volumes/c0229543f8974d87bd07caae4493ca58
>[negroni] Completed 200 OK in 598.635Âµs
>[negroni] Started GET /volumes/e6860021031e7cf362c2b3824d6c351f
>[negroni] Completed 200 OK in 566.078Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 401 Unauthorized in 121.691Âµs
>[negroni] Started POST /volumes
>[heketi] INFO 2018/06/08 07:46:17 Allocating brick set #0
>[negroni] Completed 202 Accepted in 25.34431ms
>[asynchttp] INFO 2018/06/08 07:46:17 asynchttp.go:288: Started job f453822daa8f3bafba9a9883210f917c
>[heketi] INFO 2018/06/08 07:46:17 Started async operation: Create Volume
>[negroni] Started GET /queue/f453822daa8f3bafba9a9883210f917c
>[negroni] Completed 200 OK in 118.227Âµs
>[heketi] INFO 2018/06/08 07:46:17 Creating brick 43ffab10bdb18f3bcce61f5e0c04684f
>[heketi] INFO 2018/06/08 07:46:17 Creating brick fea99506a7e983d4d4765dc1d2462625
>[heketi] INFO 2018/06/08 07:46:17 Creating brick b55f73cf47bf2a1fcb341a2e01fd79db
>[kubeexec] DEBUG 2018/06/08 07:46:17 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir -p /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_b55f73cf47bf2a1fcb341a2e01fd79db
>Result: 
>[kubeexec] DEBUG 2018/06/08 07:46:17 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir -p /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_fea99506a7e983d4d4765dc1d2462625
>Result: 
>[kubeexec] DEBUG 2018/06/08 07:46:17 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir -p /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_43ffab10bdb18f3bcce61f5e0c04684f
>Result: 
>[kubeexec] DEBUG 2018/06/08 07:46:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvcreate --autobackup=n --poolmetadatasize 53248K --chunksize 256K --size 10485760K --thin vg_b861af9cc67f5bf8d5a8f9e38f56b14e/tp_43ffab10bdb18f3bcce61f5e0c04684f --virtualsize 10485760K --name brick_43ffab10bdb18f3bcce61f5e0c04684f
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_43ffab10bdb18f3bcce61f5e0c04684f" created.
>[kubeexec] DEBUG 2018/06/08 07:46:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvcreate --autobackup=n --poolmetadatasize 53248K --chunksize 256K --size 10485760K --thin vg_3a4297677881963e3f80124971d50eea/tp_b55f73cf47bf2a1fcb341a2e01fd79db --virtualsize 10485760K --name brick_b55f73cf47bf2a1fcb341a2e01fd79db
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_b55f73cf47bf2a1fcb341a2e01fd79db" created.
>[kubeexec] DEBUG 2018/06/08 07:46:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvcreate --autobackup=n --poolmetadatasize 53248K --chunksize 256K --size 10485760K --thin vg_96f1667f2f1ced2c5ef94772922be93b/tp_fea99506a7e983d4d4765dc1d2462625 --virtualsize 10485760K --name brick_fea99506a7e983d4d4765dc1d2462625
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_fea99506a7e983d4d4765dc1d2462625" created.
>[kubeexec] DEBUG 2018/06/08 07:46:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_b55f73cf47bf2a1fcb341a2e01fd79db
>Result: meta-data=/dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_b55f73cf47bf2a1fcb341a2e01fd79db isize=512    agcount=16, agsize=163840 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=2621440, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 07:46:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_43ffab10bdb18f3bcce61f5e0c04684f
>Result: meta-data=/dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_43ffab10bdb18f3bcce61f5e0c04684f isize=512    agcount=16, agsize=163840 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=2621440, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 07:46:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_fea99506a7e983d4d4765dc1d2462625
>Result: meta-data=/dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_fea99506a7e983d4d4765dc1d2462625 isize=512    agcount=16, agsize=163840 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=2621440, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 07:46:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: awk "BEGIN {print \"/dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_43ffab10bdb18f3bcce61f5e0c04684f /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_43ffab10bdb18f3bcce61f5e0c04684f xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 07:46:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: awk "BEGIN {print \"/dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_b55f73cf47bf2a1fcb341a2e01fd79db /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_b55f73cf47bf2a1fcb341a2e01fd79db xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 07:46:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: awk "BEGIN {print \"/dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_fea99506a7e983d4d4765dc1d2462625 /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_fea99506a7e983d4d4765dc1d2462625 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 07:46:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_43ffab10bdb18f3bcce61f5e0c04684f /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_43ffab10bdb18f3bcce61f5e0c04684f
>Result: 
>[kubeexec] DEBUG 2018/06/08 07:46:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_b55f73cf47bf2a1fcb341a2e01fd79db /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_b55f73cf47bf2a1fcb341a2e01fd79db
>Result: 
>[kubeexec] DEBUG 2018/06/08 07:46:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_fea99506a7e983d4d4765dc1d2462625 /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_fea99506a7e983d4d4765dc1d2462625
>Result: 
>[kubeexec] DEBUG 2018/06/08 07:46:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_43ffab10bdb18f3bcce61f5e0c04684f/brick
>Result: 
>[negroni] Started GET /queue/f453822daa8f3bafba9a9883210f917c
>[negroni] Completed 200 OK in 99.179Âµs
>[kubeexec] DEBUG 2018/06/08 07:46:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_b55f73cf47bf2a1fcb341a2e01fd79db/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 07:46:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_fea99506a7e983d4d4765dc1d2462625/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 07:46:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: chown :2001 /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_43ffab10bdb18f3bcce61f5e0c04684f/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 07:46:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: chown :2001 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_b55f73cf47bf2a1fcb341a2e01fd79db/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 07:46:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: chown :2001 /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_fea99506a7e983d4d4765dc1d2462625/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 07:46:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: chmod 2775 /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_43ffab10bdb18f3bcce61f5e0c04684f/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 07:46:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: chmod 2775 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_b55f73cf47bf2a1fcb341a2e01fd79db/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 07:46:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: chmod 2775 /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_fea99506a7e983d4d4765dc1d2462625/brick
>Result: 
>[cmdexec] INFO 2018/06/08 07:46:18 Creating volume cns-vol_glusterfs_mongodb5_06f1079a-6af0-11e8-ab19-005056a5f18a replica 3
>[kubeexec] DEBUG 2018/06/08 07:46:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script volume create cns-vol_glusterfs_mongodb5_06f1079a-6af0-11e8-ab19-005056a5f18a replica 3 10.70.47.76:/var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_fea99506a7e983d4d4765dc1d2462625/brick 10.70.46.187:/var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_43ffab10bdb18f3bcce61f5e0c04684f/brick 10.70.46.122:/var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_b55f73cf47bf2a1fcb341a2e01fd79db/brick
>Result: volume create: cns-vol_glusterfs_mongodb5_06f1079a-6af0-11e8-ab19-005056a5f18a: success: please start the volume to access data
>[negroni] Started GET /queue/f453822daa8f3bafba9a9883210f917c
>[negroni] Completed 200 OK in 201.816Âµs
>[negroni] Started GET /queue/f453822daa8f3bafba9a9883210f917c
>[negroni] Completed 200 OK in 111.033Âµs
>[negroni] Started DELETE /volumes/c0229543f8974d87bd07caae4493ca58
>[negroni] Completed 202 Accepted in 13.399124ms
>[asynchttp] INFO 2018/06/08 07:46:20 asynchttp.go:288: Started job b2ad0d8c3ceb29259d4f9cbc0ae00dc9
>[heketi] INFO 2018/06/08 07:46:20 Started async operation: Delete Volume
>[negroni] Started GET /queue/b2ad0d8c3ceb29259d4f9cbc0ae00dc9
>[negroni] Completed 200 OK in 125.365Âµs
>[kubeexec] DEBUG 2018/06/08 07:46:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script volume start cns-vol_glusterfs_mongodb5_06f1079a-6af0-11e8-ab19-005056a5f18a
>Result: volume start: cns-vol_glusterfs_mongodb5_06f1079a-6af0-11e8-ab19-005056a5f18a: success
>[heketi] INFO 2018/06/08 07:46:21 Create Volume succeeded
>[asynchttp] INFO 2018/06/08 07:46:21 asynchttp.go:292: Completed job f453822daa8f3bafba9a9883210f917c in 3.879387245s
>[negroni] Started GET /queue/f453822daa8f3bafba9a9883210f917c
>[negroni] Completed 303 See Other in 174.248Âµs
>[negroni] Started GET /volumes/ce1dbe38a55233b6023eaf7f32109115
>[negroni] Completed 200 OK in 2.285885ms
>[negroni] Started GET /clusters/0a73c60efdd4673113b668afea101e6d
>[negroni] Completed 200 OK in 362.113Âµs
>[negroni] Started GET /nodes/278bd6b4e16a8e62ef15aaae22e6abc1
>[negroni] Completed 200 OK in 1.186475ms
>[negroni] Started GET /nodes/70423cc0bcd044fe5ba8bbbd256a3e49
>[negroni] Completed 200 OK in 1.02557ms
>[negroni] Started GET /nodes/d942fe6c0ee5691b7cc263968f97b650
>[negroni] Completed 200 OK in 755.982Âµs
>[kubeexec] DEBUG 2018/06/08 07:46:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script snapshot list cns-vol_glusterfs_mongodb5_cca32e68-6aef-11e8-ab19-005056a5f18a --xml
>Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
><cliOutput>
>  <opRet>0</opRet>
>  <opErrno>0</opErrno>
>  <opErrstr/>
>  <snapList>
>    <count>0</count>
>  </snapList>
></cliOutput>
>[negroni] Started GET /queue/b2ad0d8c3ceb29259d4f9cbc0ae00dc9
>[negroni] Completed 200 OK in 122.337Âµs
>[negroni] Started GET /queue/b2ad0d8c3ceb29259d4f9cbc0ae00dc9
>[negroni] Completed 200 OK in 154.279Âµs
>[negroni] Started GET /queue/b2ad0d8c3ceb29259d4f9cbc0ae00dc9
>[negroni] Completed 200 OK in 99.47Âµs
>[negroni] Started GET /queue/b2ad0d8c3ceb29259d4f9cbc0ae00dc9
>[negroni] Completed 200 OK in 145.777Âµs
>[negroni] Started GET /queue/b2ad0d8c3ceb29259d4f9cbc0ae00dc9
>[negroni] Completed 200 OK in 157.21Âµs
>[kubeexec] DEBUG 2018/06/08 07:46:25 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script volume stop cns-vol_glusterfs_mongodb5_cca32e68-6aef-11e8-ab19-005056a5f18a force
>Result: volume stop: cns-vol_glusterfs_mongodb5_cca32e68-6aef-11e8-ab19-005056a5f18a: success
>[kubeexec] DEBUG 2018/06/08 07:46:26 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script volume delete cns-vol_glusterfs_mongodb5_cca32e68-6aef-11e8-ab19-005056a5f18a
>Result: volume delete: cns-vol_glusterfs_mongodb5_cca32e68-6aef-11e8-ab19-005056a5f18a: success
>[heketi] INFO 2018/06/08 07:46:26 Deleting brick 6fe37e54fbf6c9bb1e66e24c971d7a76
>[heketi] INFO 2018/06/08 07:46:26 Deleting brick 8f853bf979222c0f3eec071f7605f2c2
>[heketi] INFO 2018/06/08 07:46:26 Deleting brick fa4c6b50ba3c59db3faff8234b61e970
>[kubeexec] DEBUG 2018/06/08 07:46:26 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mount | grep -w /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_8f853bf979222c0f3eec071f7605f2c2 | cut -d" " -f1
>Result: /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_8f853bf979222c0f3eec071f7605f2c2
>[kubeexec] DEBUG 2018/06/08 07:46:26 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mount | grep -w /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_6fe37e54fbf6c9bb1e66e24c971d7a76 | cut -d" " -f1
>Result: /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_6fe37e54fbf6c9bb1e66e24c971d7a76
>[kubeexec] DEBUG 2018/06/08 07:46:26 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mount | grep -w /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_fa4c6b50ba3c59db3faff8234b61e970 | cut -d" " -f1
>Result: /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_fa4c6b50ba3c59db3faff8234b61e970
>[kubeexec] DEBUG 2018/06/08 07:46:26 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_8f853bf979222c0f3eec071f7605f2c2
>
>Result:   vg_d389f0278a774bd7443a09af960961d8/tp_8f853bf979222c0f3eec071f7605f2c2
>[kubeexec] DEBUG 2018/06/08 07:46:26 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_6fe37e54fbf6c9bb1e66e24c971d7a76
>
>Result:   vg_96f1667f2f1ced2c5ef94772922be93b/tp_6fe37e54fbf6c9bb1e66e24c971d7a76
>[negroni] Started GET /queue/b2ad0d8c3ceb29259d4f9cbc0ae00dc9
>[negroni] Completed 200 OK in 128.029Âµs
>[kubeexec] DEBUG 2018/06/08 07:46:27 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_fa4c6b50ba3c59db3faff8234b61e970
>
>Result:   vg_3a4297677881963e3f80124971d50eea/tp_fa4c6b50ba3c59db3faff8234b61e970
>[kubeexec] DEBUG 2018/06/08 07:46:27 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: umount /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_8f853bf979222c0f3eec071f7605f2c2
>Result: 
>[kubeexec] DEBUG 2018/06/08 07:46:27 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: umount /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_6fe37e54fbf6c9bb1e66e24c971d7a76
>Result: 
>[negroni] Started GET /queue/b2ad0d8c3ceb29259d4f9cbc0ae00dc9
>[negroni] Completed 200 OK in 108.182Âµs
>[kubeexec] DEBUG 2018/06/08 07:46:28 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: umount /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_fa4c6b50ba3c59db3faff8234b61e970
>Result: 
>[kubeexec] DEBUG 2018/06/08 07:46:28 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: sed -i.save "/brick_8f853bf979222c0f3eec071f7605f2c2/d" /var/lib/heketi/fstab
>Result: 
>[negroni] Started GET /queue/b2ad0d8c3ceb29259d4f9cbc0ae00dc9
>[negroni] Completed 200 OK in 97.21Âµs
>[kubeexec] DEBUG 2018/06/08 07:46:28 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: sed -i.save "/brick_6fe37e54fbf6c9bb1e66e24c971d7a76/d" /var/lib/heketi/fstab
>Result: 
>[kubeexec] DEBUG 2018/06/08 07:46:29 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: sed -i.save "/brick_fa4c6b50ba3c59db3faff8234b61e970/d" /var/lib/heketi/fstab
>Result: 
>[negroni] Started GET /queue/b2ad0d8c3ceb29259d4f9cbc0ae00dc9
>[negroni] Completed 200 OK in 119.315Âµs
>[kubeexec] DEBUG 2018/06/08 07:46:29 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvremove --autobackup=n -f /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_8f853bf979222c0f3eec071f7605f2c2
>
>Result:   Logical volume "brick_8f853bf979222c0f3eec071f7605f2c2" successfully removed
>[kubeexec] DEBUG 2018/06/08 07:46:30 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvremove --autobackup=n -f /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_6fe37e54fbf6c9bb1e66e24c971d7a76
>
>Result:   Logical volume "brick_6fe37e54fbf6c9bb1e66e24c971d7a76" successfully removed
>[negroni] Started GET /queue/b2ad0d8c3ceb29259d4f9cbc0ae00dc9
>[negroni] Completed 200 OK in 114.714Âµs
>[kubeexec] DEBUG 2018/06/08 07:46:30 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvremove --autobackup=n -f /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_fa4c6b50ba3c59db3faff8234b61e970
>
>Result:   Logical volume "brick_fa4c6b50ba3c59db3faff8234b61e970" successfully removed
>[kubeexec] DEBUG 2018/06/08 07:46:31 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvs --noheadings --options=thin_count   vg_d389f0278a774bd7443a09af960961d8/tp_8f853bf979222c0f3eec071f7605f2c2
>
>Result:        0
>[kubeexec] DEBUG 2018/06/08 07:46:31 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvs --noheadings --options=thin_count   vg_96f1667f2f1ced2c5ef94772922be93b/tp_6fe37e54fbf6c9bb1e66e24c971d7a76
>
>Result:        0
>[negroni] Started GET /queue/b2ad0d8c3ceb29259d4f9cbc0ae00dc9
>[negroni] Completed 200 OK in 130.245Âµs
>[kubeexec] DEBUG 2018/06/08 07:46:32 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvs --noheadings --options=thin_count   vg_3a4297677881963e3f80124971d50eea/tp_fa4c6b50ba3c59db3faff8234b61e970
>
>Result:        0
>[kubeexec] DEBUG 2018/06/08 07:46:32 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvremove --autobackup=n -f   vg_d389f0278a774bd7443a09af960961d8/tp_8f853bf979222c0f3eec071f7605f2c2
>
>Result:   Logical volume "tp_8f853bf979222c0f3eec071f7605f2c2" successfully removed
>[kubeexec] DEBUG 2018/06/08 07:46:32 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvremove --autobackup=n -f   vg_96f1667f2f1ced2c5ef94772922be93b/tp_6fe37e54fbf6c9bb1e66e24c971d7a76
>
>Result:   Logical volume "tp_6fe37e54fbf6c9bb1e66e24c971d7a76" successfully removed
>[negroni] Started GET /queue/b2ad0d8c3ceb29259d4f9cbc0ae00dc9
>[negroni] Completed 200 OK in 103.826Âµs
>[kubeexec] DEBUG 2018/06/08 07:46:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvremove --autobackup=n -f   vg_3a4297677881963e3f80124971d50eea/tp_fa4c6b50ba3c59db3faff8234b61e970
>
>Result:   Logical volume "tp_fa4c6b50ba3c59db3faff8234b61e970" successfully removed
>[kubeexec] DEBUG 2018/06/08 07:46:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: rmdir /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_8f853bf979222c0f3eec071f7605f2c2
>Result: 
>[kubeexec] DEBUG 2018/06/08 07:46:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: rmdir /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_6fe37e54fbf6c9bb1e66e24c971d7a76
>Result: 
>[negroni] Started GET /queue/b2ad0d8c3ceb29259d4f9cbc0ae00dc9
>[negroni] Completed 200 OK in 133.981Âµs
>[kubeexec] DEBUG 2018/06/08 07:46:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: rmdir /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_fa4c6b50ba3c59db3faff8234b61e970
>Result: 
>[heketi] INFO 2018/06/08 07:46:33 Delete Volume succeeded
>[asynchttp] INFO 2018/06/08 07:46:33 asynchttp.go:292: Completed job b2ad0d8c3ceb29259d4f9cbc0ae00dc9 in 13.273784177s
>[negroni] Started GET /queue/b2ad0d8c3ceb29259d4f9cbc0ae00dc9
>[negroni] Completed 204 No Content in 158.184Âµs
>[negroni] Started DELETE /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 202 Accepted in 12.048504ms
>[asynchttp] INFO 2018/06/08 07:46:50 asynchttp.go:288: Started job c8dde8dd483c1ac657f48f19aa7fa167
>[heketi] INFO 2018/06/08 07:46:50 Started async operation: Delete Volume
>[negroni] Started GET /queue/c8dde8dd483c1ac657f48f19aa7fa167
>[negroni] Completed 200 OK in 146.932Âµs
>[kubeexec] DEBUG 2018/06/08 07:46:50 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script snapshot list vol_9c7d5f0f473cce6914804135f0b8ddcd --xml
>Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
><cliOutput>
>  <opRet>-1</opRet>
>  <opErrno>30806</opErrno>
>  <opErrstr>Volume (vol_9c7d5f0f473cce6914804135f0b8ddcd) does not exist</opErrstr>
></cliOutput>
>[kubeexec] ERROR 2018/06/08 07:46:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume stop vol_9c7d5f0f473cce6914804135f0b8ddcd force] on glusterfs-storage-gxp7c: Err[command terminated with exit code 1]: Stdout []: Stderr [volume stop: vol_9c7d5f0f473cce6914804135f0b8ddcd: failed: Volume vol_9c7d5f0f473cce6914804135f0b8ddcd does not exist
>]
>[cmdexec] ERROR 2018/06/08 07:46:51 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:144: Unable to stop volume vol_9c7d5f0f473cce6914804135f0b8ddcd: Unable to execute command on glusterfs-storage-gxp7c: volume stop: vol_9c7d5f0f473cce6914804135f0b8ddcd: failed: Volume vol_9c7d5f0f473cce6914804135f0b8ddcd does not exist
>[kubeexec] ERROR 2018/06/08 07:46:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume delete vol_9c7d5f0f473cce6914804135f0b8ddcd] on glusterfs-storage-gxp7c: Err[command terminated with exit code 1]: Stdout []: Stderr [volume delete: vol_9c7d5f0f473cce6914804135f0b8ddcd: failed: Volume vol_9c7d5f0f473cce6914804135f0b8ddcd does not exist
>]
>[cmdexec] ERROR 2018/06/08 07:46:51 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:153: Unable to delete volume vol_9c7d5f0f473cce6914804135f0b8ddcd: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c7d5f0f473cce6914804135f0b8ddcd: failed: Volume vol_9c7d5f0f473cce6914804135f0b8ddcd does not exist
>[heketi] ERROR 2018/06/08 07:46:51 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:525: Unable to delete volume: Unable to delete volume vol_9c7d5f0f473cce6914804135f0b8ddcd: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c7d5f0f473cce6914804135f0b8ddcd: failed: Volume vol_9c7d5f0f473cce6914804135f0b8ddcd does not exist
>[heketi] ERROR 2018/06/08 07:46:51 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:429: Error executing delete volume: Unable to delete volume vol_9c7d5f0f473cce6914804135f0b8ddcd: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c7d5f0f473cce6914804135f0b8ddcd: failed: Volume vol_9c7d5f0f473cce6914804135f0b8ddcd does not exist
>[heketi] ERROR 2018/06/08 07:46:51 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:1185: Delete Volume Failed: Unable to delete volume vol_9c7d5f0f473cce6914804135f0b8ddcd: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c7d5f0f473cce6914804135f0b8ddcd: failed: Volume vol_9c7d5f0f473cce6914804135f0b8ddcd does not exist
>[asynchttp] INFO 2018/06/08 07:46:51 asynchttp.go:292: Completed job c8dde8dd483c1ac657f48f19aa7fa167 in 735.20644ms
>[negroni] Started GET /queue/c8dde8dd483c1ac657f48f19aa7fa167
>[negroni] Completed 500 Internal Server Error in 126.96Âµs
>[heketi] INFO 2018/06/08 07:47:14 Starting Node Health Status refresh
>[cmdexec] INFO 2018/06/08 07:47:14 Check Glusterd service status in node dhcp46-187.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 07:47:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:33 UTC; 1h 23min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48fed567_6ae4_11e8_bc4b_005056a5f18a.slice/docker-a506d9aaca4b440d184df40d01e90b2040811c8090bf4c19d892081307b28c0c.scope/system.slice/glusterd.service
>           ââ 433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ1228 /usr/sbin/glusterfsd -s 10.70.46.187 --volfile-id heketidbstorage.10.70.46.187.var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.187-var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.pid -S /var/run/gluster/c08898c2e3b0190c62df3a9d4a013c92.socket --brick-name /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.log --xlator-option *-posix.glusterd-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ6536 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/8fb729a0bbd51b204491c603c84acaa4.socket --xlator-option *replicate*.node-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034
>
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Unit glusterd.service entered failed state.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: glusterd.service failed.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:33 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 07:47:14 Periodic health check status: node 278bd6b4e16a8e62ef15aaae22e6abc1 up=true
>[cmdexec] INFO 2018/06/08 07:47:14 Check Glusterd service status in node dhcp46-122.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 07:47:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:42 UTC; 1h 23min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4900c968_6ae4_11e8_bc4b_005056a5f18a.slice/docker-438fb5fb16a82a3e701b28fa1f1ef8623dc37ca06374c604e5781a2b2ab2e06d.scope/system.slice/glusterd.service
>           ââ 433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ1106 /usr/sbin/glusterfsd -s 10.70.46.122 --volfile-id heketidbstorage.10.70.46.122.var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.122-var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.pid -S /var/run/gluster/9fa3ac3b12826912287adc88c26ac134.socket --brick-name /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_29e5b36d921cca1b0699871d9727c932/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.log --xlator-option *-posix.glusterd-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ6351 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f8994cdb09351d83e70da08261d99c64.socket --xlator-option *replicate*.node-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0
>
>Jun 08 06:23:40 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:42 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 07:47:14 Periodic health check status: node 70423cc0bcd044fe5ba8bbbd256a3e49 up=true
>[cmdexec] INFO 2018/06/08 07:47:14 Check Glusterd service status in node dhcp47-76.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 07:47:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 07:25:20 UTC; 21min ago
>  Process: 828 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 830 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5cb2299_6aec_11e8_bc4b_005056a5f18a.slice/docker-7f9223abc9a8641cea68f374f4039235515978ee2dab970ac6cbef182c535426.scope/system.slice/glusterd.service
>           ââ 830 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ1197 /usr/sbin/glusterfsd -s 10.70.47.76 --volfile-id heketidbstorage.10.70.47.76.var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.76-var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.pid -S /var/run/gluster/b6976f9ddc3abc90e1dfa91c35a7c611.socket --brick-name /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_54c06cdf52c6fe1e3a3f8a9d39cde33d/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.log --xlator-option *-posix.glusterd-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ4123 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/ec468453b11caed5259f5e1abcc65b44.socket --xlator-option *replicate*.node-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94
>
>Jun 08 07:25:17 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 07:25:20 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 07:47:14 Periodic health check status: node d942fe6c0ee5691b7cc263968f97b650 up=true
>[heketi] INFO 2018/06/08 07:47:14 Cleaned 0 nodes from health cache
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 1.665737ms
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 1.839529ms
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 959.701Âµs
>[negroni] Started GET /volumes/ce1dbe38a55233b6023eaf7f32109115
>[negroni] Completed 200 OK in 1.579851ms
>[negroni] Started GET /volumes/e6860021031e7cf362c2b3824d6c351f
>[negroni] Completed 200 OK in 602.239Âµs
>[negroni] Started DELETE /volumes/e6860021031e7cf362c2b3824d6c351f
>[negroni] Completed 202 Accepted in 13.882575ms
>[asynchttp] INFO 2018/06/08 07:47:54 asynchttp.go:288: Started job 1b6a09cb7439ecd97f5d0426af64b3af
>[heketi] INFO 2018/06/08 07:47:54 Started async operation: Delete Volume
>[negroni] Started GET /queue/1b6a09cb7439ecd97f5d0426af64b3af
>[negroni] Completed 200 OK in 93.171Âµs
>[kubeexec] DEBUG 2018/06/08 07:47:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script snapshot list vol_e6860021031e7cf362c2b3824d6c351f --xml
>Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
><cliOutput>
>  <opRet>0</opRet>
>  <opErrno>0</opErrno>
>  <opErrstr/>
>  <snapList>
>    <count>0</count>
>  </snapList>
></cliOutput>
>[negroni] Started GET /queue/1b6a09cb7439ecd97f5d0426af64b3af
>[negroni] Completed 200 OK in 117.043Âµs
>[negroni] Started GET /queue/1b6a09cb7439ecd97f5d0426af64b3af
>[negroni] Completed 200 OK in 114.347Âµs
>[kubeexec] DEBUG 2018/06/08 07:47:56 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script volume stop vol_e6860021031e7cf362c2b3824d6c351f force
>Result: volume stop: vol_e6860021031e7cf362c2b3824d6c351f: success
>[kubeexec] DEBUG 2018/06/08 07:47:56 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script volume delete vol_e6860021031e7cf362c2b3824d6c351f
>Result: volume delete: vol_e6860021031e7cf362c2b3824d6c351f: success
>[heketi] INFO 2018/06/08 07:47:56 Deleting brick 12c589a01d51afb92169220f3344b7df
>[heketi] INFO 2018/06/08 07:47:56 Deleting brick 5174520a82f2e1505984f6e152a16133
>[heketi] INFO 2018/06/08 07:47:56 Deleting brick 548c0a969ccc1c76bea99bded13b1297
>[kubeexec] DEBUG 2018/06/08 07:47:56 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mount | grep -w /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_548c0a969ccc1c76bea99bded13b1297 | cut -d" " -f1
>Result: /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_548c0a969ccc1c76bea99bded13b1297
>[kubeexec] DEBUG 2018/06/08 07:47:56 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mount | grep -w /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_12c589a01d51afb92169220f3344b7df | cut -d" " -f1
>Result: /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_12c589a01d51afb92169220f3344b7df
>[kubeexec] DEBUG 2018/06/08 07:47:56 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mount | grep -w /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_5174520a82f2e1505984f6e152a16133 | cut -d" " -f1
>Result: /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_5174520a82f2e1505984f6e152a16133
>[negroni] Started GET /queue/1b6a09cb7439ecd97f5d0426af64b3af
>[negroni] Completed 200 OK in 139.662Âµs
>[kubeexec] DEBUG 2018/06/08 07:47:57 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_548c0a969ccc1c76bea99bded13b1297
>
>Result:   vg_b861af9cc67f5bf8d5a8f9e38f56b14e/tp_548c0a969ccc1c76bea99bded13b1297
>[kubeexec] DEBUG 2018/06/08 07:47:57 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_12c589a01d51afb92169220f3344b7df
>
>Result:   vg_9394bc70699b006c5460c9f654cf345f/tp_12c589a01d51afb92169220f3344b7df
>[kubeexec] DEBUG 2018/06/08 07:47:57 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_5174520a82f2e1505984f6e152a16133
>
>Result:   vg_3a4297677881963e3f80124971d50eea/tp_5174520a82f2e1505984f6e152a16133
>[negroni] Started GET /queue/1b6a09cb7439ecd97f5d0426af64b3af
>[negroni] Completed 200 OK in 170.554Âµs
>[kubeexec] DEBUG 2018/06/08 07:47:58 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: umount /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_548c0a969ccc1c76bea99bded13b1297
>Result: 
>[kubeexec] DEBUG 2018/06/08 07:47:58 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: umount /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_12c589a01d51afb92169220f3344b7df
>Result: 
>[kubeexec] DEBUG 2018/06/08 07:47:58 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: umount /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_5174520a82f2e1505984f6e152a16133
>Result: 
>[negroni] Started GET /queue/1b6a09cb7439ecd97f5d0426af64b3af
>[negroni] Completed 200 OK in 155.361Âµs
>[kubeexec] DEBUG 2018/06/08 07:47:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: sed -i.save "/brick_548c0a969ccc1c76bea99bded13b1297/d" /var/lib/heketi/fstab
>Result: 
>[kubeexec] DEBUG 2018/06/08 07:47:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: sed -i.save "/brick_12c589a01d51afb92169220f3344b7df/d" /var/lib/heketi/fstab
>Result: 
>[kubeexec] DEBUG 2018/06/08 07:47:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: sed -i.save "/brick_5174520a82f2e1505984f6e152a16133/d" /var/lib/heketi/fstab
>Result: 
>[negroni] Started GET /queue/1b6a09cb7439ecd97f5d0426af64b3af
>[negroni] Completed 200 OK in 101.547Âµs
>[kubeexec] DEBUG 2018/06/08 07:48:00 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvremove --autobackup=n -f /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_548c0a969ccc1c76bea99bded13b1297
>
>Result:   Logical volume "brick_548c0a969ccc1c76bea99bded13b1297" successfully removed
>[kubeexec] DEBUG 2018/06/08 07:48:00 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvremove --autobackup=n -f /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_12c589a01d51afb92169220f3344b7df
>
>Result:   Logical volume "brick_12c589a01d51afb92169220f3344b7df" successfully removed
>[negroni] Started GET /queue/1b6a09cb7439ecd97f5d0426af64b3af
>[negroni] Completed 200 OK in 119.092Âµs
>[kubeexec] DEBUG 2018/06/08 07:48:01 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvremove --autobackup=n -f /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_5174520a82f2e1505984f6e152a16133
>
>Result:   Logical volume "brick_5174520a82f2e1505984f6e152a16133" successfully removed
>[kubeexec] DEBUG 2018/06/08 07:48:01 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvs --noheadings --options=thin_count   vg_b861af9cc67f5bf8d5a8f9e38f56b14e/tp_548c0a969ccc1c76bea99bded13b1297
>
>Result:        0
>[kubeexec] DEBUG 2018/06/08 07:48:02 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvs --noheadings --options=thin_count   vg_9394bc70699b006c5460c9f654cf345f/tp_12c589a01d51afb92169220f3344b7df
>
>Result:        0
>[negroni] Started GET /queue/1b6a09cb7439ecd97f5d0426af64b3af
>[negroni] Completed 200 OK in 88.298Âµs
>[kubeexec] DEBUG 2018/06/08 07:48:02 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvs --noheadings --options=thin_count   vg_3a4297677881963e3f80124971d50eea/tp_5174520a82f2e1505984f6e152a16133
>
>Result:        0
>[negroni] Started GET /queue/1b6a09cb7439ecd97f5d0426af64b3af
>[negroni] Completed 200 OK in 173.095Âµs
>[kubeexec] DEBUG 2018/06/08 07:48:03 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvremove --autobackup=n -f   vg_b861af9cc67f5bf8d5a8f9e38f56b14e/tp_548c0a969ccc1c76bea99bded13b1297
>
>Result:   Logical volume "tp_548c0a969ccc1c76bea99bded13b1297" successfully removed
>[kubeexec] DEBUG 2018/06/08 07:48:03 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvremove --autobackup=n -f   vg_9394bc70699b006c5460c9f654cf345f/tp_12c589a01d51afb92169220f3344b7df
>
>Result:   Logical volume "tp_12c589a01d51afb92169220f3344b7df" successfully removed
>[kubeexec] DEBUG 2018/06/08 07:48:03 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvremove --autobackup=n -f   vg_3a4297677881963e3f80124971d50eea/tp_5174520a82f2e1505984f6e152a16133
>
>Result:   Logical volume "tp_5174520a82f2e1505984f6e152a16133" successfully removed
>[kubeexec] DEBUG 2018/06/08 07:48:03 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: rmdir /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_548c0a969ccc1c76bea99bded13b1297
>Result: 
>[negroni] Started GET /queue/1b6a09cb7439ecd97f5d0426af64b3af
>[negroni] Completed 200 OK in 115.613Âµs
>[kubeexec] DEBUG 2018/06/08 07:48:04 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: rmdir /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_12c589a01d51afb92169220f3344b7df
>Result: 
>[kubeexec] DEBUG 2018/06/08 07:48:04 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: rmdir /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_5174520a82f2e1505984f6e152a16133
>Result: 
>[heketi] INFO 2018/06/08 07:48:04 Delete Volume succeeded
>[asynchttp] INFO 2018/06/08 07:48:04 asynchttp.go:292: Completed job 1b6a09cb7439ecd97f5d0426af64b3af in 10.602868697s
>[negroni] Started GET /queue/1b6a09cb7439ecd97f5d0426af64b3af
>[negroni] Completed 204 No Content in 116.524Âµs
>[negroni] Started DELETE /volumes/e6860021031e7cf362c2b3824d6c351f
>[negroni] Completed 404 Not Found in 1.910083ms
>[negroni] Started DELETE /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 202 Accepted in 12.786221ms
>[asynchttp] INFO 2018/06/08 07:49:05 asynchttp.go:288: Started job fc30b9aee857144b39e7e32e0c3e98e8
>[heketi] INFO 2018/06/08 07:49:05 Started async operation: Delete Volume
>[negroni] Started GET /queue/fc30b9aee857144b39e7e32e0c3e98e8
>[negroni] Completed 200 OK in 100.497Âµs
>[kubeexec] DEBUG 2018/06/08 07:49:05 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script snapshot list vol_9c7d5f0f473cce6914804135f0b8ddcd --xml
>Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
><cliOutput>
>  <opRet>-1</opRet>
>  <opErrno>30806</opErrno>
>  <opErrstr>Volume (vol_9c7d5f0f473cce6914804135f0b8ddcd) does not exist</opErrstr>
></cliOutput>
>[kubeexec] ERROR 2018/06/08 07:49:06 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume stop vol_9c7d5f0f473cce6914804135f0b8ddcd force] on glusterfs-storage-gxp7c: Err[command terminated with exit code 1]: Stdout []: Stderr [volume stop: vol_9c7d5f0f473cce6914804135f0b8ddcd: failed: Volume vol_9c7d5f0f473cce6914804135f0b8ddcd does not exist
>]
>[cmdexec] ERROR 2018/06/08 07:49:06 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:144: Unable to stop volume vol_9c7d5f0f473cce6914804135f0b8ddcd: Unable to execute command on glusterfs-storage-gxp7c: volume stop: vol_9c7d5f0f473cce6914804135f0b8ddcd: failed: Volume vol_9c7d5f0f473cce6914804135f0b8ddcd does not exist
>[kubeexec] ERROR 2018/06/08 07:49:06 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume delete vol_9c7d5f0f473cce6914804135f0b8ddcd] on glusterfs-storage-gxp7c: Err[command terminated with exit code 1]: Stdout []: Stderr [volume delete: vol_9c7d5f0f473cce6914804135f0b8ddcd: failed: Volume vol_9c7d5f0f473cce6914804135f0b8ddcd does not exist
>]
>[cmdexec] ERROR 2018/06/08 07:49:06 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:153: Unable to delete volume vol_9c7d5f0f473cce6914804135f0b8ddcd: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c7d5f0f473cce6914804135f0b8ddcd: failed: Volume vol_9c7d5f0f473cce6914804135f0b8ddcd does not exist
>[heketi] ERROR 2018/06/08 07:49:06 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:525: Unable to delete volume: Unable to delete volume vol_9c7d5f0f473cce6914804135f0b8ddcd: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c7d5f0f473cce6914804135f0b8ddcd: failed: Volume vol_9c7d5f0f473cce6914804135f0b8ddcd does not exist
>[heketi] ERROR 2018/06/08 07:49:06 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:429: Error executing delete volume: Unable to delete volume vol_9c7d5f0f473cce6914804135f0b8ddcd: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c7d5f0f473cce6914804135f0b8ddcd: failed: Volume vol_9c7d5f0f473cce6914804135f0b8ddcd does not exist
>[asynchttp] INFO 2018/06/08 07:49:06 asynchttp.go:292: Completed job fc30b9aee857144b39e7e32e0c3e98e8 in 763.812826ms
>[heketi] ERROR 2018/06/08 07:49:06 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:1185: Delete Volume Failed: Unable to delete volume vol_9c7d5f0f473cce6914804135f0b8ddcd: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c7d5f0f473cce6914804135f0b8ddcd: failed: Volume vol_9c7d5f0f473cce6914804135f0b8ddcd does not exist
>[negroni] Started GET /queue/fc30b9aee857144b39e7e32e0c3e98e8
>[negroni] Completed 500 Internal Server Error in 159.589Âµs
>[heketi] INFO 2018/06/08 07:49:14 Starting Node Health Status refresh
>[cmdexec] INFO 2018/06/08 07:49:14 Check Glusterd service status in node dhcp46-187.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 07:49:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:33 UTC; 1h 25min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48fed567_6ae4_11e8_bc4b_005056a5f18a.slice/docker-a506d9aaca4b440d184df40d01e90b2040811c8090bf4c19d892081307b28c0c.scope/system.slice/glusterd.service
>           ââ 433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ1228 /usr/sbin/glusterfsd -s 10.70.46.187 --volfile-id heketidbstorage.10.70.46.187.var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.187-var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.pid -S /var/run/gluster/c08898c2e3b0190c62df3a9d4a013c92.socket --brick-name /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.log --xlator-option *-posix.glusterd-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ6684 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/8fb729a0bbd51b204491c603c84acaa4.socket --xlator-option *replicate*.node-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034
>
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Unit glusterd.service entered failed state.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: glusterd.service failed.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:33 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 07:49:14 Periodic health check status: node 278bd6b4e16a8e62ef15aaae22e6abc1 up=true
>[cmdexec] INFO 2018/06/08 07:49:14 Check Glusterd service status in node dhcp46-122.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 07:49:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:42 UTC; 1h 25min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4900c968_6ae4_11e8_bc4b_005056a5f18a.slice/docker-438fb5fb16a82a3e701b28fa1f1ef8623dc37ca06374c604e5781a2b2ab2e06d.scope/system.slice/glusterd.service
>           ââ 433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ1106 /usr/sbin/glusterfsd -s 10.70.46.122 --volfile-id heketidbstorage.10.70.46.122.var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.122-var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.pid -S /var/run/gluster/9fa3ac3b12826912287adc88c26ac134.socket --brick-name /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_29e5b36d921cca1b0699871d9727c932/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.log --xlator-option *-posix.glusterd-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ6500 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f8994cdb09351d83e70da08261d99c64.socket --xlator-option *replicate*.node-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0
>
>Jun 08 06:23:40 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:42 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 07:49:14 Periodic health check status: node 70423cc0bcd044fe5ba8bbbd256a3e49 up=true
>[cmdexec] INFO 2018/06/08 07:49:14 Check Glusterd service status in node dhcp47-76.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 07:49:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 07:25:20 UTC; 23min ago
>  Process: 828 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 830 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5cb2299_6aec_11e8_bc4b_005056a5f18a.slice/docker-7f9223abc9a8641cea68f374f4039235515978ee2dab970ac6cbef182c535426.scope/system.slice/glusterd.service
>           ââ 830 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ1197 /usr/sbin/glusterfsd -s 10.70.47.76 --volfile-id heketidbstorage.10.70.47.76.var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.76-var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.pid -S /var/run/gluster/b6976f9ddc3abc90e1dfa91c35a7c611.socket --brick-name /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_54c06cdf52c6fe1e3a3f8a9d39cde33d/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.log --xlator-option *-posix.glusterd-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ4372 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/ec468453b11caed5259f5e1abcc65b44.socket --xlator-option *replicate*.node-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94
>
>Jun 08 07:25:17 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 07:25:20 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 07:49:14 Periodic health check status: node d942fe6c0ee5691b7cc263968f97b650 up=true
>[heketi] INFO 2018/06/08 07:49:14 Cleaned 0 nodes from health cache
>[heketi] INFO 2018/06/08 07:51:14 Starting Node Health Status refresh
>[cmdexec] INFO 2018/06/08 07:51:14 Check Glusterd service status in node dhcp46-187.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 07:51:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:33 UTC; 1h 27min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48fed567_6ae4_11e8_bc4b_005056a5f18a.slice/docker-a506d9aaca4b440d184df40d01e90b2040811c8090bf4c19d892081307b28c0c.scope/system.slice/glusterd.service
>           ââ 433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ1228 /usr/sbin/glusterfsd -s 10.70.46.187 --volfile-id heketidbstorage.10.70.46.187.var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.187-var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.pid -S /var/run/gluster/c08898c2e3b0190c62df3a9d4a013c92.socket --brick-name /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.log --xlator-option *-posix.glusterd-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ6684 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/8fb729a0bbd51b204491c603c84acaa4.socket --xlator-option *replicate*.node-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034
>
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Unit glusterd.service entered failed state.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: glusterd.service failed.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:33 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 07:51:14 Periodic health check status: node 278bd6b4e16a8e62ef15aaae22e6abc1 up=true
>[cmdexec] INFO 2018/06/08 07:51:14 Check Glusterd service status in node dhcp46-122.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 07:51:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:42 UTC; 1h 27min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4900c968_6ae4_11e8_bc4b_005056a5f18a.slice/docker-438fb5fb16a82a3e701b28fa1f1ef8623dc37ca06374c604e5781a2b2ab2e06d.scope/system.slice/glusterd.service
>           ââ 433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ1106 /usr/sbin/glusterfsd -s 10.70.46.122 --volfile-id heketidbstorage.10.70.46.122.var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.122-var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.pid -S /var/run/gluster/9fa3ac3b12826912287adc88c26ac134.socket --brick-name /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_29e5b36d921cca1b0699871d9727c932/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.log --xlator-option *-posix.glusterd-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ6500 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f8994cdb09351d83e70da08261d99c64.socket --xlator-option *replicate*.node-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0
>
>Jun 08 06:23:40 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:42 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 07:51:14 Periodic health check status: node 70423cc0bcd044fe5ba8bbbd256a3e49 up=true
>[cmdexec] INFO 2018/06/08 07:51:14 Check Glusterd service status in node dhcp47-76.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 07:51:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 07:25:20 UTC; 25min ago
>  Process: 828 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 830 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5cb2299_6aec_11e8_bc4b_005056a5f18a.slice/docker-7f9223abc9a8641cea68f374f4039235515978ee2dab970ac6cbef182c535426.scope/system.slice/glusterd.service
>           ââ 830 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ1197 /usr/sbin/glusterfsd -s 10.70.47.76 --volfile-id heketidbstorage.10.70.47.76.var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.76-var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.pid -S /var/run/gluster/b6976f9ddc3abc90e1dfa91c35a7c611.socket --brick-name /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_54c06cdf52c6fe1e3a3f8a9d39cde33d/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.log --xlator-option *-posix.glusterd-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ4372 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/ec468453b11caed5259f5e1abcc65b44.socket --xlator-option *replicate*.node-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94
>
>Jun 08 07:25:17 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 07:25:20 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 07:51:14 Periodic health check status: node d942fe6c0ee5691b7cc263968f97b650 up=true
>[heketi] INFO 2018/06/08 07:51:14 Cleaned 0 nodes from health cache
>[negroni] Started DELETE /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 202 Accepted in 15.137299ms
>[asynchttp] INFO 2018/06/08 07:51:20 asynchttp.go:288: Started job b95907c995b6e365d88e25a41d21ca63
>[heketi] INFO 2018/06/08 07:51:20 Started async operation: Delete Volume
>[negroni] Started GET /queue/b95907c995b6e365d88e25a41d21ca63
>[negroni] Completed 200 OK in 163.558Âµs
>[kubeexec] DEBUG 2018/06/08 07:51:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script snapshot list vol_9c7d5f0f473cce6914804135f0b8ddcd --xml
>Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
><cliOutput>
>  <opRet>-1</opRet>
>  <opErrno>30806</opErrno>
>  <opErrstr>Volume (vol_9c7d5f0f473cce6914804135f0b8ddcd) does not exist</opErrstr>
></cliOutput>
>[kubeexec] ERROR 2018/06/08 07:51:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume stop vol_9c7d5f0f473cce6914804135f0b8ddcd force] on glusterfs-storage-gxp7c: Err[command terminated with exit code 1]: Stdout []: Stderr [volume stop: vol_9c7d5f0f473cce6914804135f0b8ddcd: failed: Volume vol_9c7d5f0f473cce6914804135f0b8ddcd does not exist
>]
>[cmdexec] ERROR 2018/06/08 07:51:21 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:144: Unable to stop volume vol_9c7d5f0f473cce6914804135f0b8ddcd: Unable to execute command on glusterfs-storage-gxp7c: volume stop: vol_9c7d5f0f473cce6914804135f0b8ddcd: failed: Volume vol_9c7d5f0f473cce6914804135f0b8ddcd does not exist
>[kubeexec] ERROR 2018/06/08 07:51:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume delete vol_9c7d5f0f473cce6914804135f0b8ddcd] on glusterfs-storage-gxp7c: Err[command terminated with exit code 1]: Stdout []: Stderr [volume delete: vol_9c7d5f0f473cce6914804135f0b8ddcd: failed: Volume vol_9c7d5f0f473cce6914804135f0b8ddcd does not exist
>]
>[cmdexec] ERROR 2018/06/08 07:51:21 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:153: Unable to delete volume vol_9c7d5f0f473cce6914804135f0b8ddcd: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c7d5f0f473cce6914804135f0b8ddcd: failed: Volume vol_9c7d5f0f473cce6914804135f0b8ddcd does not exist
>[heketi] ERROR 2018/06/08 07:51:21 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:525: Unable to delete volume: Unable to delete volume vol_9c7d5f0f473cce6914804135f0b8ddcd: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c7d5f0f473cce6914804135f0b8ddcd: failed: Volume vol_9c7d5f0f473cce6914804135f0b8ddcd does not exist
>[heketi] ERROR 2018/06/08 07:51:21 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:429: Error executing delete volume: Unable to delete volume vol_9c7d5f0f473cce6914804135f0b8ddcd: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c7d5f0f473cce6914804135f0b8ddcd: failed: Volume vol_9c7d5f0f473cce6914804135f0b8ddcd does not exist
>[heketi] ERROR 2018/06/08 07:51:21 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:1185: Delete Volume Failed: Unable to delete volume vol_9c7d5f0f473cce6914804135f0b8ddcd: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c7d5f0f473cce6914804135f0b8ddcd: failed: Volume vol_9c7d5f0f473cce6914804135f0b8ddcd does not exist
>[asynchttp] INFO 2018/06/08 07:51:21 asynchttp.go:292: Completed job b95907c995b6e365d88e25a41d21ca63 in 722.905951ms
>[negroni] Started GET /queue/b95907c995b6e365d88e25a41d21ca63
>[negroni] Completed 500 Internal Server Error in 136.323Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 1.692182ms
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 1.013983ms
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 546.015Âµs
>[negroni] Started GET /volumes/ce1dbe38a55233b6023eaf7f32109115
>[negroni] Completed 200 OK in 723.741Âµs
>[heketi] INFO 2018/06/08 07:53:14 Starting Node Health Status refresh
>[cmdexec] INFO 2018/06/08 07:53:14 Check Glusterd service status in node dhcp46-187.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 07:53:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:33 UTC; 1h 29min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48fed567_6ae4_11e8_bc4b_005056a5f18a.slice/docker-a506d9aaca4b440d184df40d01e90b2040811c8090bf4c19d892081307b28c0c.scope/system.slice/glusterd.service
>           ââ 433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ1228 /usr/sbin/glusterfsd -s 10.70.46.187 --volfile-id heketidbstorage.10.70.46.187.var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.187-var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.pid -S /var/run/gluster/c08898c2e3b0190c62df3a9d4a013c92.socket --brick-name /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.log --xlator-option *-posix.glusterd-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ6684 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/8fb729a0bbd51b204491c603c84acaa4.socket --xlator-option *replicate*.node-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034
>
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Unit glusterd.service entered failed state.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: glusterd.service failed.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:33 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 07:53:14 Periodic health check status: node 278bd6b4e16a8e62ef15aaae22e6abc1 up=true
>[cmdexec] INFO 2018/06/08 07:53:14 Check Glusterd service status in node dhcp46-122.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 07:53:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:42 UTC; 1h 29min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4900c968_6ae4_11e8_bc4b_005056a5f18a.slice/docker-438fb5fb16a82a3e701b28fa1f1ef8623dc37ca06374c604e5781a2b2ab2e06d.scope/system.slice/glusterd.service
>           ââ 433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ1106 /usr/sbin/glusterfsd -s 10.70.46.122 --volfile-id heketidbstorage.10.70.46.122.var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.122-var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.pid -S /var/run/gluster/9fa3ac3b12826912287adc88c26ac134.socket --brick-name /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_29e5b36d921cca1b0699871d9727c932/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.log --xlator-option *-posix.glusterd-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ6500 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f8994cdb09351d83e70da08261d99c64.socket --xlator-option *replicate*.node-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0
>
>Jun 08 06:23:40 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:42 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 07:53:14 Periodic health check status: node 70423cc0bcd044fe5ba8bbbd256a3e49 up=true
>[cmdexec] INFO 2018/06/08 07:53:14 Check Glusterd service status in node dhcp47-76.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 07:53:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 07:25:20 UTC; 27min ago
>  Process: 828 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 830 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5cb2299_6aec_11e8_bc4b_005056a5f18a.slice/docker-7f9223abc9a8641cea68f374f4039235515978ee2dab970ac6cbef182c535426.scope/system.slice/glusterd.service
>           ââ 830 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ1197 /usr/sbin/glusterfsd -s 10.70.47.76 --volfile-id heketidbstorage.10.70.47.76.var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.76-var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.pid -S /var/run/gluster/b6976f9ddc3abc90e1dfa91c35a7c611.socket --brick-name /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_54c06cdf52c6fe1e3a3f8a9d39cde33d/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.log --xlator-option *-posix.glusterd-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ4372 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/ec468453b11caed5259f5e1abcc65b44.socket --xlator-option *replicate*.node-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94
>
>Jun 08 07:25:17 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 07:25:20 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 07:53:14 Periodic health check status: node d942fe6c0ee5691b7cc263968f97b650 up=true
>[heketi] INFO 2018/06/08 07:53:14 Cleaned 0 nodes from health cache
>[negroni] Started DELETE /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 202 Accepted in 10.184128ms
>[asynchttp] INFO 2018/06/08 07:53:35 asynchttp.go:288: Started job ca499bed75730c5d49dfa3c30b8bfb4d
>[heketi] INFO 2018/06/08 07:53:35 Started async operation: Delete Volume
>[negroni] Started GET /queue/ca499bed75730c5d49dfa3c30b8bfb4d
>[negroni] Completed 200 OK in 199.139Âµs
>[kubeexec] DEBUG 2018/06/08 07:53:35 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script snapshot list vol_9c7d5f0f473cce6914804135f0b8ddcd --xml
>Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
><cliOutput>
>  <opRet>-1</opRet>
>  <opErrno>30806</opErrno>
>  <opErrstr>Volume (vol_9c7d5f0f473cce6914804135f0b8ddcd) does not exist</opErrstr>
></cliOutput>
>[kubeexec] ERROR 2018/06/08 07:53:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume stop vol_9c7d5f0f473cce6914804135f0b8ddcd force] on glusterfs-storage-gxp7c: Err[command terminated with exit code 1]: Stdout []: Stderr [volume stop: vol_9c7d5f0f473cce6914804135f0b8ddcd: failed: Volume vol_9c7d5f0f473cce6914804135f0b8ddcd does not exist
>]
>[cmdexec] ERROR 2018/06/08 07:53:36 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:144: Unable to stop volume vol_9c7d5f0f473cce6914804135f0b8ddcd: Unable to execute command on glusterfs-storage-gxp7c: volume stop: vol_9c7d5f0f473cce6914804135f0b8ddcd: failed: Volume vol_9c7d5f0f473cce6914804135f0b8ddcd does not exist
>[kubeexec] ERROR 2018/06/08 07:53:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume delete vol_9c7d5f0f473cce6914804135f0b8ddcd] on glusterfs-storage-gxp7c: Err[command terminated with exit code 1]: Stdout []: Stderr [volume delete: vol_9c7d5f0f473cce6914804135f0b8ddcd: failed: Volume vol_9c7d5f0f473cce6914804135f0b8ddcd does not exist
>]
>[cmdexec] ERROR 2018/06/08 07:53:36 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:153: Unable to delete volume vol_9c7d5f0f473cce6914804135f0b8ddcd: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c7d5f0f473cce6914804135f0b8ddcd: failed: Volume vol_9c7d5f0f473cce6914804135f0b8ddcd does not exist
>[heketi] ERROR 2018/06/08 07:53:36 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:525: Unable to delete volume: Unable to delete volume vol_9c7d5f0f473cce6914804135f0b8ddcd: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c7d5f0f473cce6914804135f0b8ddcd: failed: Volume vol_9c7d5f0f473cce6914804135f0b8ddcd does not exist
>[heketi] ERROR 2018/06/08 07:53:36 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:429: Error executing delete volume: Unable to delete volume vol_9c7d5f0f473cce6914804135f0b8ddcd: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c7d5f0f473cce6914804135f0b8ddcd: failed: Volume vol_9c7d5f0f473cce6914804135f0b8ddcd does not exist
>[heketi] ERROR 2018/06/08 07:53:36 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:1185: Delete Volume Failed: Unable to delete volume vol_9c7d5f0f473cce6914804135f0b8ddcd: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c7d5f0f473cce6914804135f0b8ddcd: failed: Volume vol_9c7d5f0f473cce6914804135f0b8ddcd does not exist
>[asynchttp] INFO 2018/06/08 07:53:36 asynchttp.go:292: Completed job ca499bed75730c5d49dfa3c30b8bfb4d in 766.156461ms
>[negroni] Started GET /queue/ca499bed75730c5d49dfa3c30b8bfb4d
>[negroni] Completed 500 Internal Server Error in 136.705Âµs
>[heketi] INFO 2018/06/08 07:55:14 Starting Node Health Status refresh
>[cmdexec] INFO 2018/06/08 07:55:14 Check Glusterd service status in node dhcp46-187.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 07:55:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:33 UTC; 1h 31min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48fed567_6ae4_11e8_bc4b_005056a5f18a.slice/docker-a506d9aaca4b440d184df40d01e90b2040811c8090bf4c19d892081307b28c0c.scope/system.slice/glusterd.service
>           ââ 433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ1228 /usr/sbin/glusterfsd -s 10.70.46.187 --volfile-id heketidbstorage.10.70.46.187.var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.187-var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.pid -S /var/run/gluster/c08898c2e3b0190c62df3a9d4a013c92.socket --brick-name /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.log --xlator-option *-posix.glusterd-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ6684 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/8fb729a0bbd51b204491c603c84acaa4.socket --xlator-option *replicate*.node-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034
>
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Unit glusterd.service entered failed state.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: glusterd.service failed.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:33 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 07:55:14 Periodic health check status: node 278bd6b4e16a8e62ef15aaae22e6abc1 up=true
>[cmdexec] INFO 2018/06/08 07:55:14 Check Glusterd service status in node dhcp46-122.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 07:55:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:42 UTC; 1h 31min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4900c968_6ae4_11e8_bc4b_005056a5f18a.slice/docker-438fb5fb16a82a3e701b28fa1f1ef8623dc37ca06374c604e5781a2b2ab2e06d.scope/system.slice/glusterd.service
>           ââ 433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ1106 /usr/sbin/glusterfsd -s 10.70.46.122 --volfile-id heketidbstorage.10.70.46.122.var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.122-var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.pid -S /var/run/gluster/9fa3ac3b12826912287adc88c26ac134.socket --brick-name /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_29e5b36d921cca1b0699871d9727c932/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.log --xlator-option *-posix.glusterd-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ6500 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f8994cdb09351d83e70da08261d99c64.socket --xlator-option *replicate*.node-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0
>
>Jun 08 06:23:40 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:42 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 07:55:14 Periodic health check status: node 70423cc0bcd044fe5ba8bbbd256a3e49 up=true
>[cmdexec] INFO 2018/06/08 07:55:14 Check Glusterd service status in node dhcp47-76.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 07:55:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 07:25:20 UTC; 29min ago
>  Process: 828 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 830 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5cb2299_6aec_11e8_bc4b_005056a5f18a.slice/docker-7f9223abc9a8641cea68f374f4039235515978ee2dab970ac6cbef182c535426.scope/system.slice/glusterd.service
>           ââ 830 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ1197 /usr/sbin/glusterfsd -s 10.70.47.76 --volfile-id heketidbstorage.10.70.47.76.var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.76-var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.pid -S /var/run/gluster/b6976f9ddc3abc90e1dfa91c35a7c611.socket --brick-name /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_54c06cdf52c6fe1e3a3f8a9d39cde33d/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.log --xlator-option *-posix.glusterd-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ4372 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/ec468453b11caed5259f5e1abcc65b44.socket --xlator-option *replicate*.node-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94
>
>Jun 08 07:25:17 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 07:25:20 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 07:55:14 Periodic health check status: node d942fe6c0ee5691b7cc263968f97b650 up=true
>[heketi] INFO 2018/06/08 07:55:14 Cleaned 0 nodes from health cache
>[negroni] Started DELETE /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 202 Accepted in 11.759497ms
>[asynchttp] INFO 2018/06/08 07:55:50 asynchttp.go:288: Started job a965fa1cad76deaabfae107bc4fa7b61
>[heketi] INFO 2018/06/08 07:55:50 Started async operation: Delete Volume
>[negroni] Started GET /queue/a965fa1cad76deaabfae107bc4fa7b61
>[negroni] Completed 200 OK in 123.67Âµs
>[kubeexec] DEBUG 2018/06/08 07:55:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script snapshot list vol_9c7d5f0f473cce6914804135f0b8ddcd --xml
>Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
><cliOutput>
>  <opRet>-1</opRet>
>  <opErrno>30806</opErrno>
>  <opErrstr>Volume (vol_9c7d5f0f473cce6914804135f0b8ddcd) does not exist</opErrstr>
></cliOutput>
>[kubeexec] ERROR 2018/06/08 07:55:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume stop vol_9c7d5f0f473cce6914804135f0b8ddcd force] on glusterfs-storage-gxp7c: Err[command terminated with exit code 1]: Stdout []: Stderr [volume stop: vol_9c7d5f0f473cce6914804135f0b8ddcd: failed: Volume vol_9c7d5f0f473cce6914804135f0b8ddcd does not exist
>]
>[cmdexec] ERROR 2018/06/08 07:55:51 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:144: Unable to stop volume vol_9c7d5f0f473cce6914804135f0b8ddcd: Unable to execute command on glusterfs-storage-gxp7c: volume stop: vol_9c7d5f0f473cce6914804135f0b8ddcd: failed: Volume vol_9c7d5f0f473cce6914804135f0b8ddcd does not exist
>[kubeexec] ERROR 2018/06/08 07:55:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume delete vol_9c7d5f0f473cce6914804135f0b8ddcd] on glusterfs-storage-gxp7c: Err[command terminated with exit code 1]: Stdout []: Stderr [volume delete: vol_9c7d5f0f473cce6914804135f0b8ddcd: failed: Volume vol_9c7d5f0f473cce6914804135f0b8ddcd does not exist
>]
>[cmdexec] ERROR 2018/06/08 07:55:51 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:153: Unable to delete volume vol_9c7d5f0f473cce6914804135f0b8ddcd: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c7d5f0f473cce6914804135f0b8ddcd: failed: Volume vol_9c7d5f0f473cce6914804135f0b8ddcd does not exist
>[heketi] ERROR 2018/06/08 07:55:51 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:525: Unable to delete volume: Unable to delete volume vol_9c7d5f0f473cce6914804135f0b8ddcd: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c7d5f0f473cce6914804135f0b8ddcd: failed: Volume vol_9c7d5f0f473cce6914804135f0b8ddcd does not exist
>[heketi] ERROR 2018/06/08 07:55:51 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:429: Error executing delete volume: Unable to delete volume vol_9c7d5f0f473cce6914804135f0b8ddcd: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c7d5f0f473cce6914804135f0b8ddcd: failed: Volume vol_9c7d5f0f473cce6914804135f0b8ddcd does not exist
>[asynchttp] INFO 2018/06/08 07:55:51 asynchttp.go:292: Completed job a965fa1cad76deaabfae107bc4fa7b61 in 966.874938ms
>[heketi] ERROR 2018/06/08 07:55:51 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:1185: Delete Volume Failed: Unable to delete volume vol_9c7d5f0f473cce6914804135f0b8ddcd: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c7d5f0f473cce6914804135f0b8ddcd: failed: Volume vol_9c7d5f0f473cce6914804135f0b8ddcd does not exist
>[negroni] Started GET /queue/a965fa1cad76deaabfae107bc4fa7b61
>[negroni] Completed 500 Internal Server Error in 190.476Âµs
>[heketi] INFO 2018/06/08 07:57:14 Starting Node Health Status refresh
>[cmdexec] INFO 2018/06/08 07:57:14 Check Glusterd service status in node dhcp46-187.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 07:57:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:33 UTC; 1h 33min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48fed567_6ae4_11e8_bc4b_005056a5f18a.slice/docker-a506d9aaca4b440d184df40d01e90b2040811c8090bf4c19d892081307b28c0c.scope/system.slice/glusterd.service
>           ââ 433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ1228 /usr/sbin/glusterfsd -s 10.70.46.187 --volfile-id heketidbstorage.10.70.46.187.var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.187-var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.pid -S /var/run/gluster/c08898c2e3b0190c62df3a9d4a013c92.socket --brick-name /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.log --xlator-option *-posix.glusterd-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ6684 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/8fb729a0bbd51b204491c603c84acaa4.socket --xlator-option *replicate*.node-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034
>
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Unit glusterd.service entered failed state.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: glusterd.service failed.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:33 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 07:57:14 Periodic health check status: node 278bd6b4e16a8e62ef15aaae22e6abc1 up=true
>[cmdexec] INFO 2018/06/08 07:57:14 Check Glusterd service status in node dhcp46-122.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 07:57:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:42 UTC; 1h 33min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4900c968_6ae4_11e8_bc4b_005056a5f18a.slice/docker-438fb5fb16a82a3e701b28fa1f1ef8623dc37ca06374c604e5781a2b2ab2e06d.scope/system.slice/glusterd.service
>           ââ 433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ1106 /usr/sbin/glusterfsd -s 10.70.46.122 --volfile-id heketidbstorage.10.70.46.122.var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.122-var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.pid -S /var/run/gluster/9fa3ac3b12826912287adc88c26ac134.socket --brick-name /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_29e5b36d921cca1b0699871d9727c932/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.log --xlator-option *-posix.glusterd-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ6500 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f8994cdb09351d83e70da08261d99c64.socket --xlator-option *replicate*.node-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0
>
>Jun 08 06:23:40 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:42 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 07:57:14 Periodic health check status: node 70423cc0bcd044fe5ba8bbbd256a3e49 up=true
>[cmdexec] INFO 2018/06/08 07:57:14 Check Glusterd service status in node dhcp47-76.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 07:57:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 07:25:20 UTC; 31min ago
>  Process: 828 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 830 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5cb2299_6aec_11e8_bc4b_005056a5f18a.slice/docker-7f9223abc9a8641cea68f374f4039235515978ee2dab970ac6cbef182c535426.scope/system.slice/glusterd.service
>           ââ 830 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ1197 /usr/sbin/glusterfsd -s 10.70.47.76 --volfile-id heketidbstorage.10.70.47.76.var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.76-var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.pid -S /var/run/gluster/b6976f9ddc3abc90e1dfa91c35a7c611.socket --brick-name /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_54c06cdf52c6fe1e3a3f8a9d39cde33d/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.log --xlator-option *-posix.glusterd-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ4372 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/ec468453b11caed5259f5e1abcc65b44.socket --xlator-option *replicate*.node-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94
>
>Jun 08 07:25:17 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 07:25:20 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 07:57:14 Periodic health check status: node d942fe6c0ee5691b7cc263968f97b650 up=true
>[heketi] INFO 2018/06/08 07:57:14 Cleaned 0 nodes from health cache
>[negroni] Started DELETE /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 202 Accepted in 10.438114ms
>[asynchttp] INFO 2018/06/08 07:57:53 asynchttp.go:288: Started job 2440346aa2f29a060d86e6947457eae1
>[heketi] INFO 2018/06/08 07:57:53 Started async operation: Delete Volume
>[negroni] Started GET /queue/2440346aa2f29a060d86e6947457eae1
>[negroni] Completed 200 OK in 132.135Âµs
>[kubeexec] DEBUG 2018/06/08 07:57:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script snapshot list vol_9c7d5f0f473cce6914804135f0b8ddcd --xml
>Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
><cliOutput>
>  <opRet>-1</opRet>
>  <opErrno>30806</opErrno>
>  <opErrstr>Volume (vol_9c7d5f0f473cce6914804135f0b8ddcd) does not exist</opErrstr>
></cliOutput>
>[kubeexec] ERROR 2018/06/08 07:57:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume stop vol_9c7d5f0f473cce6914804135f0b8ddcd force] on glusterfs-storage-gxp7c: Err[command terminated with exit code 1]: Stdout []: Stderr [volume stop: vol_9c7d5f0f473cce6914804135f0b8ddcd: failed: Volume vol_9c7d5f0f473cce6914804135f0b8ddcd does not exist
>]
>[cmdexec] ERROR 2018/06/08 07:57:54 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:144: Unable to stop volume vol_9c7d5f0f473cce6914804135f0b8ddcd: Unable to execute command on glusterfs-storage-gxp7c: volume stop: vol_9c7d5f0f473cce6914804135f0b8ddcd: failed: Volume vol_9c7d5f0f473cce6914804135f0b8ddcd does not exist
>[kubeexec] ERROR 2018/06/08 07:57:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume delete vol_9c7d5f0f473cce6914804135f0b8ddcd] on glusterfs-storage-gxp7c: Err[command terminated with exit code 1]: Stdout []: Stderr [volume delete: vol_9c7d5f0f473cce6914804135f0b8ddcd: failed: Volume vol_9c7d5f0f473cce6914804135f0b8ddcd does not exist
>]
>[cmdexec] ERROR 2018/06/08 07:57:54 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:153: Unable to delete volume vol_9c7d5f0f473cce6914804135f0b8ddcd: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c7d5f0f473cce6914804135f0b8ddcd: failed: Volume vol_9c7d5f0f473cce6914804135f0b8ddcd does not exist
>[heketi] ERROR 2018/06/08 07:57:54 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:525: Unable to delete volume: Unable to delete volume vol_9c7d5f0f473cce6914804135f0b8ddcd: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c7d5f0f473cce6914804135f0b8ddcd: failed: Volume vol_9c7d5f0f473cce6914804135f0b8ddcd does not exist
>[heketi] ERROR 2018/06/08 07:57:54 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:429: Error executing delete volume: Unable to delete volume vol_9c7d5f0f473cce6914804135f0b8ddcd: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c7d5f0f473cce6914804135f0b8ddcd: failed: Volume vol_9c7d5f0f473cce6914804135f0b8ddcd does not exist
>[asynchttp] INFO 2018/06/08 07:57:54 asynchttp.go:292: Completed job 2440346aa2f29a060d86e6947457eae1 in 736.716267ms
>[heketi] ERROR 2018/06/08 07:57:54 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:1185: Delete Volume Failed: Unable to delete volume vol_9c7d5f0f473cce6914804135f0b8ddcd: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c7d5f0f473cce6914804135f0b8ddcd: failed: Volume vol_9c7d5f0f473cce6914804135f0b8ddcd does not exist
>[negroni] Started GET /queue/2440346aa2f29a060d86e6947457eae1
>[negroni] Completed 500 Internal Server Error in 234.005Âµs
>[heketi] INFO 2018/06/08 07:59:14 Starting Node Health Status refresh
>[cmdexec] INFO 2018/06/08 07:59:14 Check Glusterd service status in node dhcp46-187.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 07:59:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:33 UTC; 1h 35min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48fed567_6ae4_11e8_bc4b_005056a5f18a.slice/docker-a506d9aaca4b440d184df40d01e90b2040811c8090bf4c19d892081307b28c0c.scope/system.slice/glusterd.service
>           ââ 433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ1228 /usr/sbin/glusterfsd -s 10.70.46.187 --volfile-id heketidbstorage.10.70.46.187.var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.187-var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.pid -S /var/run/gluster/c08898c2e3b0190c62df3a9d4a013c92.socket --brick-name /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.log --xlator-option *-posix.glusterd-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ6684 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/8fb729a0bbd51b204491c603c84acaa4.socket --xlator-option *replicate*.node-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034
>
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Unit glusterd.service entered failed state.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: glusterd.service failed.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:33 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 07:59:14 Periodic health check status: node 278bd6b4e16a8e62ef15aaae22e6abc1 up=true
>[cmdexec] INFO 2018/06/08 07:59:14 Check Glusterd service status in node dhcp46-122.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 07:59:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:42 UTC; 1h 35min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4900c968_6ae4_11e8_bc4b_005056a5f18a.slice/docker-438fb5fb16a82a3e701b28fa1f1ef8623dc37ca06374c604e5781a2b2ab2e06d.scope/system.slice/glusterd.service
>           ââ 433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ1106 /usr/sbin/glusterfsd -s 10.70.46.122 --volfile-id heketidbstorage.10.70.46.122.var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.122-var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.pid -S /var/run/gluster/9fa3ac3b12826912287adc88c26ac134.socket --brick-name /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_29e5b36d921cca1b0699871d9727c932/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.log --xlator-option *-posix.glusterd-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ6500 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f8994cdb09351d83e70da08261d99c64.socket --xlator-option *replicate*.node-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0
>
>Jun 08 06:23:40 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:42 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 07:59:14 Periodic health check status: node 70423cc0bcd044fe5ba8bbbd256a3e49 up=true
>[cmdexec] INFO 2018/06/08 07:59:14 Check Glusterd service status in node dhcp47-76.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 07:59:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 07:25:20 UTC; 33min ago
>  Process: 828 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 830 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5cb2299_6aec_11e8_bc4b_005056a5f18a.slice/docker-7f9223abc9a8641cea68f374f4039235515978ee2dab970ac6cbef182c535426.scope/system.slice/glusterd.service
>           ââ 830 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ1197 /usr/sbin/glusterfsd -s 10.70.47.76 --volfile-id heketidbstorage.10.70.47.76.var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.76-var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.pid -S /var/run/gluster/b6976f9ddc3abc90e1dfa91c35a7c611.socket --brick-name /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_54c06cdf52c6fe1e3a3f8a9d39cde33d/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.log --xlator-option *-posix.glusterd-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ4372 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/ec468453b11caed5259f5e1abcc65b44.socket --xlator-option *replicate*.node-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94
>
>Jun 08 07:25:17 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 07:25:20 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 07:59:14 Periodic health check status: node d942fe6c0ee5691b7cc263968f97b650 up=true
>[heketi] INFO 2018/06/08 07:59:14 Cleaned 0 nodes from health cache
>[negroni] Started DELETE /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 202 Accepted in 13.008059ms
>[asynchttp] INFO 2018/06/08 08:00:05 asynchttp.go:288: Started job 44604b026004cedca0e24a37c3c85d1d
>[heketi] INFO 2018/06/08 08:00:05 Started async operation: Delete Volume
>[negroni] Started GET /queue/44604b026004cedca0e24a37c3c85d1d
>[negroni] Completed 200 OK in 130.721Âµs
>[kubeexec] DEBUG 2018/06/08 08:00:05 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script snapshot list vol_9c7d5f0f473cce6914804135f0b8ddcd --xml
>Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
><cliOutput>
>  <opRet>-1</opRet>
>  <opErrno>30806</opErrno>
>  <opErrstr>Volume (vol_9c7d5f0f473cce6914804135f0b8ddcd) does not exist</opErrstr>
></cliOutput>
>[kubeexec] ERROR 2018/06/08 08:00:06 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume stop vol_9c7d5f0f473cce6914804135f0b8ddcd force] on glusterfs-storage-gxp7c: Err[command terminated with exit code 1]: Stdout []: Stderr [volume stop: vol_9c7d5f0f473cce6914804135f0b8ddcd: failed: Volume vol_9c7d5f0f473cce6914804135f0b8ddcd does not exist
>]
>[cmdexec] ERROR 2018/06/08 08:00:06 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:144: Unable to stop volume vol_9c7d5f0f473cce6914804135f0b8ddcd: Unable to execute command on glusterfs-storage-gxp7c: volume stop: vol_9c7d5f0f473cce6914804135f0b8ddcd: failed: Volume vol_9c7d5f0f473cce6914804135f0b8ddcd does not exist
>[kubeexec] ERROR 2018/06/08 08:00:06 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume delete vol_9c7d5f0f473cce6914804135f0b8ddcd] on glusterfs-storage-gxp7c: Err[command terminated with exit code 1]: Stdout []: Stderr [volume delete: vol_9c7d5f0f473cce6914804135f0b8ddcd: failed: Volume vol_9c7d5f0f473cce6914804135f0b8ddcd does not exist
>]
>[cmdexec] ERROR 2018/06/08 08:00:06 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:153: Unable to delete volume vol_9c7d5f0f473cce6914804135f0b8ddcd: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c7d5f0f473cce6914804135f0b8ddcd: failed: Volume vol_9c7d5f0f473cce6914804135f0b8ddcd does not exist
>[heketi] ERROR 2018/06/08 08:00:06 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:525: Unable to delete volume: Unable to delete volume vol_9c7d5f0f473cce6914804135f0b8ddcd: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c7d5f0f473cce6914804135f0b8ddcd: failed: Volume vol_9c7d5f0f473cce6914804135f0b8ddcd does not exist
>[heketi] ERROR 2018/06/08 08:00:06 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:429: Error executing delete volume: Unable to delete volume vol_9c7d5f0f473cce6914804135f0b8ddcd: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c7d5f0f473cce6914804135f0b8ddcd: failed: Volume vol_9c7d5f0f473cce6914804135f0b8ddcd does not exist
>[heketi] ERROR 2018/06/08 08:00:06 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:1185: Delete Volume Failed: Unable to delete volume vol_9c7d5f0f473cce6914804135f0b8ddcd: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c7d5f0f473cce6914804135f0b8ddcd: failed: Volume vol_9c7d5f0f473cce6914804135f0b8ddcd does not exist
>[asynchttp] INFO 2018/06/08 08:00:06 asynchttp.go:292: Completed job 44604b026004cedca0e24a37c3c85d1d in 705.536523ms
>[negroni] Started GET /queue/44604b026004cedca0e24a37c3c85d1d
>[negroni] Completed 500 Internal Server Error in 168.756Âµs
>[heketi] INFO 2018/06/08 08:01:14 Starting Node Health Status refresh
>[cmdexec] INFO 2018/06/08 08:01:14 Check Glusterd service status in node dhcp46-187.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 08:01:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:33 UTC; 1h 37min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48fed567_6ae4_11e8_bc4b_005056a5f18a.slice/docker-a506d9aaca4b440d184df40d01e90b2040811c8090bf4c19d892081307b28c0c.scope/system.slice/glusterd.service
>           ââ 433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ1228 /usr/sbin/glusterfsd -s 10.70.46.187 --volfile-id heketidbstorage.10.70.46.187.var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.187-var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.pid -S /var/run/gluster/c08898c2e3b0190c62df3a9d4a013c92.socket --brick-name /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.log --xlator-option *-posix.glusterd-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ6684 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/8fb729a0bbd51b204491c603c84acaa4.socket --xlator-option *replicate*.node-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034
>
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Unit glusterd.service entered failed state.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: glusterd.service failed.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:33 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 08:01:14 Periodic health check status: node 278bd6b4e16a8e62ef15aaae22e6abc1 up=true
>[cmdexec] INFO 2018/06/08 08:01:14 Check Glusterd service status in node dhcp46-122.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 08:01:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:42 UTC; 1h 37min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4900c968_6ae4_11e8_bc4b_005056a5f18a.slice/docker-438fb5fb16a82a3e701b28fa1f1ef8623dc37ca06374c604e5781a2b2ab2e06d.scope/system.slice/glusterd.service
>           ââ 433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ1106 /usr/sbin/glusterfsd -s 10.70.46.122 --volfile-id heketidbstorage.10.70.46.122.var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.122-var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.pid -S /var/run/gluster/9fa3ac3b12826912287adc88c26ac134.socket --brick-name /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_29e5b36d921cca1b0699871d9727c932/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.log --xlator-option *-posix.glusterd-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ6500 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f8994cdb09351d83e70da08261d99c64.socket --xlator-option *replicate*.node-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0
>
>Jun 08 06:23:40 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:42 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 08:01:14 Periodic health check status: node 70423cc0bcd044fe5ba8bbbd256a3e49 up=true
>[cmdexec] INFO 2018/06/08 08:01:14 Check Glusterd service status in node dhcp47-76.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 08:01:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 07:25:20 UTC; 35min ago
>  Process: 828 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 830 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5cb2299_6aec_11e8_bc4b_005056a5f18a.slice/docker-7f9223abc9a8641cea68f374f4039235515978ee2dab970ac6cbef182c535426.scope/system.slice/glusterd.service
>           ââ 830 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ1197 /usr/sbin/glusterfsd -s 10.70.47.76 --volfile-id heketidbstorage.10.70.47.76.var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.76-var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.pid -S /var/run/gluster/b6976f9ddc3abc90e1dfa91c35a7c611.socket --brick-name /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_54c06cdf52c6fe1e3a3f8a9d39cde33d/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.log --xlator-option *-posix.glusterd-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ4372 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/ec468453b11caed5259f5e1abcc65b44.socket --xlator-option *replicate*.node-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94
>
>Jun 08 07:25:17 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 07:25:20 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 08:01:14 Periodic health check status: node d942fe6c0ee5691b7cc263968f97b650 up=true
>[heketi] INFO 2018/06/08 08:01:14 Cleaned 0 nodes from health cache
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 1.137353ms
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 1.542438ms
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 1.070897ms
>[negroni] Started GET /volumes/ce1dbe38a55233b6023eaf7f32109115
>[negroni] Completed 200 OK in 931.32Âµs
>[negroni] Started DELETE /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 202 Accepted in 13.344579ms
>[asynchttp] INFO 2018/06/08 08:02:20 asynchttp.go:288: Started job 73d6a98d4b88a1e1e5aecd323e54ea99
>[heketi] INFO 2018/06/08 08:02:20 Started async operation: Delete Volume
>[negroni] Started GET /queue/73d6a98d4b88a1e1e5aecd323e54ea99
>[negroni] Completed 200 OK in 133.641Âµs
>[kubeexec] DEBUG 2018/06/08 08:02:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script snapshot list vol_9c7d5f0f473cce6914804135f0b8ddcd --xml
>Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
><cliOutput>
>  <opRet>-1</opRet>
>  <opErrno>30806</opErrno>
>  <opErrstr>Volume (vol_9c7d5f0f473cce6914804135f0b8ddcd) does not exist</opErrstr>
></cliOutput>
>[kubeexec] ERROR 2018/06/08 08:02:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume stop vol_9c7d5f0f473cce6914804135f0b8ddcd force] on glusterfs-storage-gxp7c: Err[command terminated with exit code 1]: Stdout []: Stderr [volume stop: vol_9c7d5f0f473cce6914804135f0b8ddcd: failed: Volume vol_9c7d5f0f473cce6914804135f0b8ddcd does not exist
>]
>[cmdexec] ERROR 2018/06/08 08:02:21 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:144: Unable to stop volume vol_9c7d5f0f473cce6914804135f0b8ddcd: Unable to execute command on glusterfs-storage-gxp7c: volume stop: vol_9c7d5f0f473cce6914804135f0b8ddcd: failed: Volume vol_9c7d5f0f473cce6914804135f0b8ddcd does not exist
>[kubeexec] ERROR 2018/06/08 08:02:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume delete vol_9c7d5f0f473cce6914804135f0b8ddcd] on glusterfs-storage-gxp7c: Err[command terminated with exit code 1]: Stdout []: Stderr [volume delete: vol_9c7d5f0f473cce6914804135f0b8ddcd: failed: Volume vol_9c7d5f0f473cce6914804135f0b8ddcd does not exist
>]
>[cmdexec] ERROR 2018/06/08 08:02:21 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:153: Unable to delete volume vol_9c7d5f0f473cce6914804135f0b8ddcd: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c7d5f0f473cce6914804135f0b8ddcd: failed: Volume vol_9c7d5f0f473cce6914804135f0b8ddcd does not exist
>[heketi] ERROR 2018/06/08 08:02:21 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:525: Unable to delete volume: Unable to delete volume vol_9c7d5f0f473cce6914804135f0b8ddcd: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c7d5f0f473cce6914804135f0b8ddcd: failed: Volume vol_9c7d5f0f473cce6914804135f0b8ddcd does not exist
>[heketi] ERROR 2018/06/08 08:02:21 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:429: Error executing delete volume: Unable to delete volume vol_9c7d5f0f473cce6914804135f0b8ddcd: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c7d5f0f473cce6914804135f0b8ddcd: failed: Volume vol_9c7d5f0f473cce6914804135f0b8ddcd does not exist
>[heketi] ERROR 2018/06/08 08:02:21 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:1185: Delete Volume Failed: Unable to delete volume vol_9c7d5f0f473cce6914804135f0b8ddcd: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c7d5f0f473cce6914804135f0b8ddcd: failed: Volume vol_9c7d5f0f473cce6914804135f0b8ddcd does not exist
>[asynchttp] INFO 2018/06/08 08:02:21 asynchttp.go:292: Completed job 73d6a98d4b88a1e1e5aecd323e54ea99 in 726.731787ms
>[negroni] Started GET /queue/73d6a98d4b88a1e1e5aecd323e54ea99
>[negroni] Completed 500 Internal Server Error in 194.858Âµs
>[heketi] INFO 2018/06/08 08:03:14 Starting Node Health Status refresh
>[cmdexec] INFO 2018/06/08 08:03:14 Check Glusterd service status in node dhcp46-187.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 08:03:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:33 UTC; 1h 39min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48fed567_6ae4_11e8_bc4b_005056a5f18a.slice/docker-a506d9aaca4b440d184df40d01e90b2040811c8090bf4c19d892081307b28c0c.scope/system.slice/glusterd.service
>           ââ 433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ1228 /usr/sbin/glusterfsd -s 10.70.46.187 --volfile-id heketidbstorage.10.70.46.187.var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.187-var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.pid -S /var/run/gluster/c08898c2e3b0190c62df3a9d4a013c92.socket --brick-name /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.log --xlator-option *-posix.glusterd-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ6684 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/8fb729a0bbd51b204491c603c84acaa4.socket --xlator-option *replicate*.node-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034
>
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Unit glusterd.service entered failed state.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: glusterd.service failed.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:33 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 08:03:14 Periodic health check status: node 278bd6b4e16a8e62ef15aaae22e6abc1 up=true
>[cmdexec] INFO 2018/06/08 08:03:14 Check Glusterd service status in node dhcp46-122.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 08:03:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:42 UTC; 1h 39min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4900c968_6ae4_11e8_bc4b_005056a5f18a.slice/docker-438fb5fb16a82a3e701b28fa1f1ef8623dc37ca06374c604e5781a2b2ab2e06d.scope/system.slice/glusterd.service
>           ââ 433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ1106 /usr/sbin/glusterfsd -s 10.70.46.122 --volfile-id heketidbstorage.10.70.46.122.var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.122-var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.pid -S /var/run/gluster/9fa3ac3b12826912287adc88c26ac134.socket --brick-name /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_29e5b36d921cca1b0699871d9727c932/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.log --xlator-option *-posix.glusterd-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ6500 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f8994cdb09351d83e70da08261d99c64.socket --xlator-option *replicate*.node-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0
>
>Jun 08 06:23:40 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:42 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 08:03:14 Periodic health check status: node 70423cc0bcd044fe5ba8bbbd256a3e49 up=true
>[cmdexec] INFO 2018/06/08 08:03:14 Check Glusterd service status in node dhcp47-76.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 08:03:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 07:25:20 UTC; 37min ago
>  Process: 828 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 830 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5cb2299_6aec_11e8_bc4b_005056a5f18a.slice/docker-7f9223abc9a8641cea68f374f4039235515978ee2dab970ac6cbef182c535426.scope/system.slice/glusterd.service
>           ââ 830 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ1197 /usr/sbin/glusterfsd -s 10.70.47.76 --volfile-id heketidbstorage.10.70.47.76.var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.76-var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.pid -S /var/run/gluster/b6976f9ddc3abc90e1dfa91c35a7c611.socket --brick-name /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_54c06cdf52c6fe1e3a3f8a9d39cde33d/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.log --xlator-option *-posix.glusterd-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ4372 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/ec468453b11caed5259f5e1abcc65b44.socket --xlator-option *replicate*.node-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94
>
>Jun 08 07:25:17 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 07:25:20 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 08:03:14 Periodic health check status: node d942fe6c0ee5691b7cc263968f97b650 up=true
>[heketi] INFO 2018/06/08 08:03:14 Cleaned 0 nodes from health cache
>[negroni] Started DELETE /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 202 Accepted in 10.883657ms
>[asynchttp] INFO 2018/06/08 08:04:35 asynchttp.go:288: Started job 9980b8714b71143cdf92265379393a9b
>[heketi] INFO 2018/06/08 08:04:35 Started async operation: Delete Volume
>[negroni] Started GET /queue/9980b8714b71143cdf92265379393a9b
>[negroni] Completed 200 OK in 178.262Âµs
>[kubeexec] DEBUG 2018/06/08 08:04:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script snapshot list vol_9c7d5f0f473cce6914804135f0b8ddcd --xml
>Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
><cliOutput>
>  <opRet>-1</opRet>
>  <opErrno>30806</opErrno>
>  <opErrstr>Volume (vol_9c7d5f0f473cce6914804135f0b8ddcd) does not exist</opErrstr>
></cliOutput>
>[kubeexec] ERROR 2018/06/08 08:04:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume stop vol_9c7d5f0f473cce6914804135f0b8ddcd force] on glusterfs-storage-gxp7c: Err[command terminated with exit code 1]: Stdout []: Stderr [volume stop: vol_9c7d5f0f473cce6914804135f0b8ddcd: failed: Volume vol_9c7d5f0f473cce6914804135f0b8ddcd does not exist
>]
>[cmdexec] ERROR 2018/06/08 08:04:36 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:144: Unable to stop volume vol_9c7d5f0f473cce6914804135f0b8ddcd: Unable to execute command on glusterfs-storage-gxp7c: volume stop: vol_9c7d5f0f473cce6914804135f0b8ddcd: failed: Volume vol_9c7d5f0f473cce6914804135f0b8ddcd does not exist
>[kubeexec] ERROR 2018/06/08 08:04:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume delete vol_9c7d5f0f473cce6914804135f0b8ddcd] on glusterfs-storage-gxp7c: Err[command terminated with exit code 1]: Stdout []: Stderr [volume delete: vol_9c7d5f0f473cce6914804135f0b8ddcd: failed: Volume vol_9c7d5f0f473cce6914804135f0b8ddcd does not exist
>]
>[cmdexec] ERROR 2018/06/08 08:04:36 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:153: Unable to delete volume vol_9c7d5f0f473cce6914804135f0b8ddcd: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c7d5f0f473cce6914804135f0b8ddcd: failed: Volume vol_9c7d5f0f473cce6914804135f0b8ddcd does not exist
>[heketi] ERROR 2018/06/08 08:04:36 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:525: Unable to delete volume: Unable to delete volume vol_9c7d5f0f473cce6914804135f0b8ddcd: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c7d5f0f473cce6914804135f0b8ddcd: failed: Volume vol_9c7d5f0f473cce6914804135f0b8ddcd does not exist
>[heketi] ERROR 2018/06/08 08:04:36 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:429: Error executing delete volume: Unable to delete volume vol_9c7d5f0f473cce6914804135f0b8ddcd: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c7d5f0f473cce6914804135f0b8ddcd: failed: Volume vol_9c7d5f0f473cce6914804135f0b8ddcd does not exist
>[asynchttp] INFO 2018/06/08 08:04:36 asynchttp.go:292: Completed job 9980b8714b71143cdf92265379393a9b in 799.612198ms
>[heketi] ERROR 2018/06/08 08:04:36 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:1185: Delete Volume Failed: Unable to delete volume vol_9c7d5f0f473cce6914804135f0b8ddcd: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c7d5f0f473cce6914804135f0b8ddcd: failed: Volume vol_9c7d5f0f473cce6914804135f0b8ddcd does not exist
>[negroni] Started GET /queue/9980b8714b71143cdf92265379393a9b
>[negroni] Completed 500 Internal Server Error in 198.788Âµs
>[heketi] INFO 2018/06/08 08:05:14 Starting Node Health Status refresh
>[cmdexec] INFO 2018/06/08 08:05:14 Check Glusterd service status in node dhcp46-187.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 08:05:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:33 UTC; 1h 41min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48fed567_6ae4_11e8_bc4b_005056a5f18a.slice/docker-a506d9aaca4b440d184df40d01e90b2040811c8090bf4c19d892081307b28c0c.scope/system.slice/glusterd.service
>           ââ 433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ1228 /usr/sbin/glusterfsd -s 10.70.46.187 --volfile-id heketidbstorage.10.70.46.187.var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.187-var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.pid -S /var/run/gluster/c08898c2e3b0190c62df3a9d4a013c92.socket --brick-name /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.log --xlator-option *-posix.glusterd-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ6684 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/8fb729a0bbd51b204491c603c84acaa4.socket --xlator-option *replicate*.node-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034
>
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Unit glusterd.service entered failed state.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: glusterd.service failed.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:33 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 08:05:14 Periodic health check status: node 278bd6b4e16a8e62ef15aaae22e6abc1 up=true
>[cmdexec] INFO 2018/06/08 08:05:14 Check Glusterd service status in node dhcp46-122.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 08:05:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:42 UTC; 1h 41min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4900c968_6ae4_11e8_bc4b_005056a5f18a.slice/docker-438fb5fb16a82a3e701b28fa1f1ef8623dc37ca06374c604e5781a2b2ab2e06d.scope/system.slice/glusterd.service
>           ââ 433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ1106 /usr/sbin/glusterfsd -s 10.70.46.122 --volfile-id heketidbstorage.10.70.46.122.var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.122-var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.pid -S /var/run/gluster/9fa3ac3b12826912287adc88c26ac134.socket --brick-name /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_29e5b36d921cca1b0699871d9727c932/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.log --xlator-option *-posix.glusterd-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ6500 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f8994cdb09351d83e70da08261d99c64.socket --xlator-option *replicate*.node-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0
>
>Jun 08 06:23:40 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:42 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 08:05:14 Periodic health check status: node 70423cc0bcd044fe5ba8bbbd256a3e49 up=true
>[cmdexec] INFO 2018/06/08 08:05:14 Check Glusterd service status in node dhcp47-76.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 08:05:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 07:25:20 UTC; 39min ago
>  Process: 828 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 830 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5cb2299_6aec_11e8_bc4b_005056a5f18a.slice/docker-7f9223abc9a8641cea68f374f4039235515978ee2dab970ac6cbef182c535426.scope/system.slice/glusterd.service
>           ââ 830 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ1197 /usr/sbin/glusterfsd -s 10.70.47.76 --volfile-id heketidbstorage.10.70.47.76.var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.76-var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.pid -S /var/run/gluster/b6976f9ddc3abc90e1dfa91c35a7c611.socket --brick-name /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_54c06cdf52c6fe1e3a3f8a9d39cde33d/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.log --xlator-option *-posix.glusterd-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ4372 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/ec468453b11caed5259f5e1abcc65b44.socket --xlator-option *replicate*.node-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94
>
>Jun 08 07:25:17 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 07:25:20 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 08:05:14 Periodic health check status: node d942fe6c0ee5691b7cc263968f97b650 up=true
>[heketi] INFO 2018/06/08 08:05:14 Cleaned 0 nodes from health cache
>[negroni] Started DELETE /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 202 Accepted in 15.146222ms
>[asynchttp] INFO 2018/06/08 08:06:50 asynchttp.go:288: Started job 80fcb26a5e686e710fcc63cfb55397ab
>[heketi] INFO 2018/06/08 08:06:50 Started async operation: Delete Volume
>[negroni] Started GET /queue/80fcb26a5e686e710fcc63cfb55397ab
>[negroni] Completed 200 OK in 144.774Âµs
>[kubeexec] DEBUG 2018/06/08 08:06:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script snapshot list vol_9c7d5f0f473cce6914804135f0b8ddcd --xml
>Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
><cliOutput>
>  <opRet>-1</opRet>
>  <opErrno>30806</opErrno>
>  <opErrstr>Volume (vol_9c7d5f0f473cce6914804135f0b8ddcd) does not exist</opErrstr>
></cliOutput>
>[kubeexec] ERROR 2018/06/08 08:06:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume stop vol_9c7d5f0f473cce6914804135f0b8ddcd force] on glusterfs-storage-gxp7c: Err[command terminated with exit code 1]: Stdout []: Stderr [volume stop: vol_9c7d5f0f473cce6914804135f0b8ddcd: failed: Volume vol_9c7d5f0f473cce6914804135f0b8ddcd does not exist
>]
>[cmdexec] ERROR 2018/06/08 08:06:51 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:144: Unable to stop volume vol_9c7d5f0f473cce6914804135f0b8ddcd: Unable to execute command on glusterfs-storage-gxp7c: volume stop: vol_9c7d5f0f473cce6914804135f0b8ddcd: failed: Volume vol_9c7d5f0f473cce6914804135f0b8ddcd does not exist
>[negroni] Started GET /queue/80fcb26a5e686e710fcc63cfb55397ab
>[negroni] Completed 200 OK in 117.128Âµs
>[kubeexec] ERROR 2018/06/08 08:06:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume delete vol_9c7d5f0f473cce6914804135f0b8ddcd] on glusterfs-storage-gxp7c: Err[command terminated with exit code 1]: Stdout []: Stderr [volume delete: vol_9c7d5f0f473cce6914804135f0b8ddcd: failed: Volume vol_9c7d5f0f473cce6914804135f0b8ddcd does not exist
>]
>[cmdexec] ERROR 2018/06/08 08:06:51 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:153: Unable to delete volume vol_9c7d5f0f473cce6914804135f0b8ddcd: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c7d5f0f473cce6914804135f0b8ddcd: failed: Volume vol_9c7d5f0f473cce6914804135f0b8ddcd does not exist
>[heketi] ERROR 2018/06/08 08:06:51 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:525: Unable to delete volume: Unable to delete volume vol_9c7d5f0f473cce6914804135f0b8ddcd: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c7d5f0f473cce6914804135f0b8ddcd: failed: Volume vol_9c7d5f0f473cce6914804135f0b8ddcd does not exist
>[heketi] ERROR 2018/06/08 08:06:51 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:429: Error executing delete volume: Unable to delete volume vol_9c7d5f0f473cce6914804135f0b8ddcd: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c7d5f0f473cce6914804135f0b8ddcd: failed: Volume vol_9c7d5f0f473cce6914804135f0b8ddcd does not exist
>[heketi] ERROR 2018/06/08 08:06:51 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:1185: Delete Volume Failed: Unable to delete volume vol_9c7d5f0f473cce6914804135f0b8ddcd: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c7d5f0f473cce6914804135f0b8ddcd: failed: Volume vol_9c7d5f0f473cce6914804135f0b8ddcd does not exist
>[asynchttp] INFO 2018/06/08 08:06:51 asynchttp.go:292: Completed job 80fcb26a5e686e710fcc63cfb55397ab in 1.054589539s
>[negroni] Started GET /queue/80fcb26a5e686e710fcc63cfb55397ab
>[negroni] Completed 500 Internal Server Error in 179.595Âµs
>[heketi] INFO 2018/06/08 08:07:14 Starting Node Health Status refresh
>[cmdexec] INFO 2018/06/08 08:07:14 Check Glusterd service status in node dhcp46-187.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 08:07:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:33 UTC; 1h 43min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48fed567_6ae4_11e8_bc4b_005056a5f18a.slice/docker-a506d9aaca4b440d184df40d01e90b2040811c8090bf4c19d892081307b28c0c.scope/system.slice/glusterd.service
>           ââ 433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ1228 /usr/sbin/glusterfsd -s 10.70.46.187 --volfile-id heketidbstorage.10.70.46.187.var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.187-var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.pid -S /var/run/gluster/c08898c2e3b0190c62df3a9d4a013c92.socket --brick-name /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.log --xlator-option *-posix.glusterd-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ6684 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/8fb729a0bbd51b204491c603c84acaa4.socket --xlator-option *replicate*.node-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034
>
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Unit glusterd.service entered failed state.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: glusterd.service failed.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:33 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 08:07:14 Periodic health check status: node 278bd6b4e16a8e62ef15aaae22e6abc1 up=true
>[cmdexec] INFO 2018/06/08 08:07:14 Check Glusterd service status in node dhcp46-122.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 08:07:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:42 UTC; 1h 43min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4900c968_6ae4_11e8_bc4b_005056a5f18a.slice/docker-438fb5fb16a82a3e701b28fa1f1ef8623dc37ca06374c604e5781a2b2ab2e06d.scope/system.slice/glusterd.service
>           ââ 433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ1106 /usr/sbin/glusterfsd -s 10.70.46.122 --volfile-id heketidbstorage.10.70.46.122.var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.122-var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.pid -S /var/run/gluster/9fa3ac3b12826912287adc88c26ac134.socket --brick-name /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_29e5b36d921cca1b0699871d9727c932/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.log --xlator-option *-posix.glusterd-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ6500 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f8994cdb09351d83e70da08261d99c64.socket --xlator-option *replicate*.node-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0
>
>Jun 08 06:23:40 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:42 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 08:07:14 Periodic health check status: node 70423cc0bcd044fe5ba8bbbd256a3e49 up=true
>[cmdexec] INFO 2018/06/08 08:07:14 Check Glusterd service status in node dhcp47-76.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 08:07:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 07:25:20 UTC; 41min ago
>  Process: 828 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 830 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5cb2299_6aec_11e8_bc4b_005056a5f18a.slice/docker-7f9223abc9a8641cea68f374f4039235515978ee2dab970ac6cbef182c535426.scope/system.slice/glusterd.service
>           ââ 830 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ1197 /usr/sbin/glusterfsd -s 10.70.47.76 --volfile-id heketidbstorage.10.70.47.76.var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.76-var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.pid -S /var/run/gluster/b6976f9ddc3abc90e1dfa91c35a7c611.socket --brick-name /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_54c06cdf52c6fe1e3a3f8a9d39cde33d/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.log --xlator-option *-posix.glusterd-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ4372 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/ec468453b11caed5259f5e1abcc65b44.socket --xlator-option *replicate*.node-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94
>
>Jun 08 07:25:17 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 07:25:20 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 08:07:14 Periodic health check status: node d942fe6c0ee5691b7cc263968f97b650 up=true
>[heketi] INFO 2018/06/08 08:07:14 Cleaned 0 nodes from health cache
>[negroni] Started DELETE /volumes/ce1dbe38a55233b6023eaf7f32109115
>[negroni] Completed 202 Accepted in 9.472215ms
>[asynchttp] INFO 2018/06/08 08:07:25 asynchttp.go:288: Started job d89e7c5134f900ef5b3960b9b88d8e39
>[heketi] INFO 2018/06/08 08:07:25 Started async operation: Delete Volume
>[negroni] Started GET /queue/d89e7c5134f900ef5b3960b9b88d8e39
>[negroni] Completed 200 OK in 130.307Âµs
>[kubeexec] DEBUG 2018/06/08 08:07:26 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: gluster --mode=script snapshot list cns-vol_glusterfs_mongodb5_06f1079a-6af0-11e8-ab19-005056a5f18a --xml
>Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
><cliOutput>
>  <opRet>0</opRet>
>  <opErrno>0</opErrno>
>  <opErrstr/>
>  <snapList>
>    <count>0</count>
>  </snapList>
></cliOutput>
>[negroni] Started GET /queue/d89e7c5134f900ef5b3960b9b88d8e39
>[negroni] Completed 200 OK in 174.306Âµs
>[negroni] Started GET /queue/d89e7c5134f900ef5b3960b9b88d8e39
>[negroni] Completed 200 OK in 158.175Âµs
>[kubeexec] DEBUG 2018/06/08 08:07:28 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: gluster --mode=script volume stop cns-vol_glusterfs_mongodb5_06f1079a-6af0-11e8-ab19-005056a5f18a force
>Result: volume stop: cns-vol_glusterfs_mongodb5_06f1079a-6af0-11e8-ab19-005056a5f18a: success
>[kubeexec] DEBUG 2018/06/08 08:07:28 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: gluster --mode=script volume delete cns-vol_glusterfs_mongodb5_06f1079a-6af0-11e8-ab19-005056a5f18a
>Result: volume delete: cns-vol_glusterfs_mongodb5_06f1079a-6af0-11e8-ab19-005056a5f18a: success
>[heketi] INFO 2018/06/08 08:07:28 Deleting brick 43ffab10bdb18f3bcce61f5e0c04684f
>[heketi] INFO 2018/06/08 08:07:28 Deleting brick fea99506a7e983d4d4765dc1d2462625
>[heketi] INFO 2018/06/08 08:07:28 Deleting brick b55f73cf47bf2a1fcb341a2e01fd79db
>[kubeexec] DEBUG 2018/06/08 08:07:28 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mount | grep -w /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_fea99506a7e983d4d4765dc1d2462625 | cut -d" " -f1
>Result: /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_fea99506a7e983d4d4765dc1d2462625
>[kubeexec] DEBUG 2018/06/08 08:07:28 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mount | grep -w /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_43ffab10bdb18f3bcce61f5e0c04684f | cut -d" " -f1
>Result: /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_43ffab10bdb18f3bcce61f5e0c04684f
>[kubeexec] DEBUG 2018/06/08 08:07:28 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mount | grep -w /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_b55f73cf47bf2a1fcb341a2e01fd79db | cut -d" " -f1
>Result: /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_b55f73cf47bf2a1fcb341a2e01fd79db
>[negroni] Started GET /queue/d89e7c5134f900ef5b3960b9b88d8e39
>[negroni] Completed 200 OK in 105.121Âµs
>[kubeexec] DEBUG 2018/06/08 08:07:28 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_fea99506a7e983d4d4765dc1d2462625
>
>Result:   vg_96f1667f2f1ced2c5ef94772922be93b/tp_fea99506a7e983d4d4765dc1d2462625
>[kubeexec] DEBUG 2018/06/08 08:07:29 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_43ffab10bdb18f3bcce61f5e0c04684f
>
>Result:   vg_b861af9cc67f5bf8d5a8f9e38f56b14e/tp_43ffab10bdb18f3bcce61f5e0c04684f
>[kubeexec] DEBUG 2018/06/08 08:07:29 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_b55f73cf47bf2a1fcb341a2e01fd79db
>
>Result:   vg_3a4297677881963e3f80124971d50eea/tp_b55f73cf47bf2a1fcb341a2e01fd79db
>[negroni] Started GET /queue/d89e7c5134f900ef5b3960b9b88d8e39
>[negroni] Completed 200 OK in 89.225Âµs
>[kubeexec] DEBUG 2018/06/08 08:07:30 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: umount /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_fea99506a7e983d4d4765dc1d2462625
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:07:30 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: umount /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_43ffab10bdb18f3bcce61f5e0c04684f
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:07:30 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: umount /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_b55f73cf47bf2a1fcb341a2e01fd79db
>Result: 
>[negroni] Started GET /queue/d89e7c5134f900ef5b3960b9b88d8e39
>[negroni] Completed 200 OK in 99.773Âµs
>[kubeexec] DEBUG 2018/06/08 08:07:31 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: sed -i.save "/brick_fea99506a7e983d4d4765dc1d2462625/d" /var/lib/heketi/fstab
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:07:31 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: sed -i.save "/brick_43ffab10bdb18f3bcce61f5e0c04684f/d" /var/lib/heketi/fstab
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:07:31 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: sed -i.save "/brick_b55f73cf47bf2a1fcb341a2e01fd79db/d" /var/lib/heketi/fstab
>Result: 
>[negroni] Started GET /queue/d89e7c5134f900ef5b3960b9b88d8e39
>[negroni] Completed 200 OK in 105.204Âµs
>[kubeexec] DEBUG 2018/06/08 08:07:32 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvremove --autobackup=n -f /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_fea99506a7e983d4d4765dc1d2462625
>
>Result:   Logical volume "brick_fea99506a7e983d4d4765dc1d2462625" successfully removed
>[kubeexec] DEBUG 2018/06/08 08:07:32 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvremove --autobackup=n -f /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_43ffab10bdb18f3bcce61f5e0c04684f
>
>Result:   Logical volume "brick_43ffab10bdb18f3bcce61f5e0c04684f" successfully removed
>[kubeexec] DEBUG 2018/06/08 08:07:32 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvremove --autobackup=n -f /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_b55f73cf47bf2a1fcb341a2e01fd79db
>
>Result:   Logical volume "brick_b55f73cf47bf2a1fcb341a2e01fd79db" successfully removed
>[negroni] Started GET /queue/d89e7c5134f900ef5b3960b9b88d8e39
>[negroni] Completed 200 OK in 206.335Âµs
>[kubeexec] DEBUG 2018/06/08 08:07:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvs --noheadings --options=thin_count   vg_96f1667f2f1ced2c5ef94772922be93b/tp_fea99506a7e983d4d4765dc1d2462625
>
>Result:        0
>[kubeexec] DEBUG 2018/06/08 08:07:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvs --noheadings --options=thin_count   vg_b861af9cc67f5bf8d5a8f9e38f56b14e/tp_43ffab10bdb18f3bcce61f5e0c04684f
>
>Result:        0
>[negroni] Started GET /queue/d89e7c5134f900ef5b3960b9b88d8e39
>[negroni] Completed 200 OK in 95.455Âµs
>[kubeexec] DEBUG 2018/06/08 08:07:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvs --noheadings --options=thin_count   vg_3a4297677881963e3f80124971d50eea/tp_b55f73cf47bf2a1fcb341a2e01fd79db
>
>Result:        0
>[kubeexec] DEBUG 2018/06/08 08:07:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvremove --autobackup=n -f   vg_96f1667f2f1ced2c5ef94772922be93b/tp_fea99506a7e983d4d4765dc1d2462625
>
>Result:   Logical volume "tp_fea99506a7e983d4d4765dc1d2462625" successfully removed
>[negroni] Started GET /queue/d89e7c5134f900ef5b3960b9b88d8e39
>[negroni] Completed 200 OK in 137.179Âµs
>[kubeexec] DEBUG 2018/06/08 08:07:35 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvremove --autobackup=n -f   vg_b861af9cc67f5bf8d5a8f9e38f56b14e/tp_43ffab10bdb18f3bcce61f5e0c04684f
>
>Result:   Logical volume "tp_43ffab10bdb18f3bcce61f5e0c04684f" successfully removed
>[kubeexec] DEBUG 2018/06/08 08:07:35 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvremove --autobackup=n -f   vg_3a4297677881963e3f80124971d50eea/tp_b55f73cf47bf2a1fcb341a2e01fd79db
>
>Result:   Logical volume "tp_b55f73cf47bf2a1fcb341a2e01fd79db" successfully removed
>[negroni] Started GET /queue/d89e7c5134f900ef5b3960b9b88d8e39
>[negroni] Completed 200 OK in 92.652Âµs
>[kubeexec] DEBUG 2018/06/08 08:07:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: rmdir /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_fea99506a7e983d4d4765dc1d2462625
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:07:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: rmdir /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_43ffab10bdb18f3bcce61f5e0c04684f
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:07:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: rmdir /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_b55f73cf47bf2a1fcb341a2e01fd79db
>Result: 
>[heketi] INFO 2018/06/08 08:07:36 Delete Volume succeeded
>[asynchttp] INFO 2018/06/08 08:07:36 asynchttp.go:292: Completed job d89e7c5134f900ef5b3960b9b88d8e39 in 10.534818093s
>[negroni] Started GET /queue/d89e7c5134f900ef5b3960b9b88d8e39
>[negroni] Completed 204 No Content in 141.43Âµs
>[heketi] INFO 2018/06/08 08:09:14 Starting Node Health Status refresh
>[cmdexec] INFO 2018/06/08 08:09:14 Check Glusterd service status in node dhcp46-187.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 08:09:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:33 UTC; 1h 45min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48fed567_6ae4_11e8_bc4b_005056a5f18a.slice/docker-a506d9aaca4b440d184df40d01e90b2040811c8090bf4c19d892081307b28c0c.scope/system.slice/glusterd.service
>           ââ 433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ1228 /usr/sbin/glusterfsd -s 10.70.46.187 --volfile-id heketidbstorage.10.70.46.187.var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.187-var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.pid -S /var/run/gluster/c08898c2e3b0190c62df3a9d4a013c92.socket --brick-name /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.log --xlator-option *-posix.glusterd-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ7384 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/8fb729a0bbd51b204491c603c84acaa4.socket --xlator-option *replicate*.node-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034
>
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Unit glusterd.service entered failed state.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: glusterd.service failed.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:33 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 08:09:14 Periodic health check status: node 278bd6b4e16a8e62ef15aaae22e6abc1 up=true
>[cmdexec] INFO 2018/06/08 08:09:14 Check Glusterd service status in node dhcp46-122.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 08:09:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:42 UTC; 1h 45min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4900c968_6ae4_11e8_bc4b_005056a5f18a.slice/docker-438fb5fb16a82a3e701b28fa1f1ef8623dc37ca06374c604e5781a2b2ab2e06d.scope/system.slice/glusterd.service
>           ââ 433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ1106 /usr/sbin/glusterfsd -s 10.70.46.122 --volfile-id heketidbstorage.10.70.46.122.var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.122-var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.pid -S /var/run/gluster/9fa3ac3b12826912287adc88c26ac134.socket --brick-name /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_29e5b36d921cca1b0699871d9727c932/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.log --xlator-option *-posix.glusterd-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ7156 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f8994cdb09351d83e70da08261d99c64.socket --xlator-option *replicate*.node-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0
>
>Jun 08 06:23:40 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:42 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 08:09:14 Periodic health check status: node 70423cc0bcd044fe5ba8bbbd256a3e49 up=true
>[cmdexec] INFO 2018/06/08 08:09:14 Check Glusterd service status in node dhcp47-76.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 08:09:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 07:25:20 UTC; 43min ago
>  Process: 828 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 830 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5cb2299_6aec_11e8_bc4b_005056a5f18a.slice/docker-7f9223abc9a8641cea68f374f4039235515978ee2dab970ac6cbef182c535426.scope/system.slice/glusterd.service
>           ââ 830 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ1197 /usr/sbin/glusterfsd -s 10.70.47.76 --volfile-id heketidbstorage.10.70.47.76.var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.76-var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.pid -S /var/run/gluster/b6976f9ddc3abc90e1dfa91c35a7c611.socket --brick-name /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_54c06cdf52c6fe1e3a3f8a9d39cde33d/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.log --xlator-option *-posix.glusterd-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ5492 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/ec468453b11caed5259f5e1abcc65b44.socket --xlator-option *replicate*.node-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94
>
>Jun 08 07:25:17 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 07:25:20 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 08:09:14 Periodic health check status: node d942fe6c0ee5691b7cc263968f97b650 up=true
>[heketi] INFO 2018/06/08 08:09:14 Cleaned 0 nodes from health cache
>[heketi] INFO 2018/06/08 08:11:14 Starting Node Health Status refresh
>[cmdexec] INFO 2018/06/08 08:11:14 Check Glusterd service status in node dhcp46-187.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 08:11:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:33 UTC; 1h 47min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48fed567_6ae4_11e8_bc4b_005056a5f18a.slice/docker-a506d9aaca4b440d184df40d01e90b2040811c8090bf4c19d892081307b28c0c.scope/system.slice/glusterd.service
>           ââ 433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ1228 /usr/sbin/glusterfsd -s 10.70.46.187 --volfile-id heketidbstorage.10.70.46.187.var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.187-var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.pid -S /var/run/gluster/c08898c2e3b0190c62df3a9d4a013c92.socket --brick-name /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.log --xlator-option *-posix.glusterd-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ7384 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/8fb729a0bbd51b204491c603c84acaa4.socket --xlator-option *replicate*.node-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034
>
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Unit glusterd.service entered failed state.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: glusterd.service failed.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:33 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 08:11:14 Periodic health check status: node 278bd6b4e16a8e62ef15aaae22e6abc1 up=true
>[cmdexec] INFO 2018/06/08 08:11:14 Check Glusterd service status in node dhcp46-122.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 08:11:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:42 UTC; 1h 47min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4900c968_6ae4_11e8_bc4b_005056a5f18a.slice/docker-438fb5fb16a82a3e701b28fa1f1ef8623dc37ca06374c604e5781a2b2ab2e06d.scope/system.slice/glusterd.service
>           ââ 433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ1106 /usr/sbin/glusterfsd -s 10.70.46.122 --volfile-id heketidbstorage.10.70.46.122.var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.122-var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.pid -S /var/run/gluster/9fa3ac3b12826912287adc88c26ac134.socket --brick-name /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_29e5b36d921cca1b0699871d9727c932/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.log --xlator-option *-posix.glusterd-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ7156 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f8994cdb09351d83e70da08261d99c64.socket --xlator-option *replicate*.node-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0
>
>Jun 08 06:23:40 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:42 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 08:11:14 Periodic health check status: node 70423cc0bcd044fe5ba8bbbd256a3e49 up=true
>[cmdexec] INFO 2018/06/08 08:11:14 Check Glusterd service status in node dhcp47-76.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 08:11:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 07:25:20 UTC; 45min ago
>  Process: 828 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 830 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5cb2299_6aec_11e8_bc4b_005056a5f18a.slice/docker-7f9223abc9a8641cea68f374f4039235515978ee2dab970ac6cbef182c535426.scope/system.slice/glusterd.service
>           ââ 830 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ1197 /usr/sbin/glusterfsd -s 10.70.47.76 --volfile-id heketidbstorage.10.70.47.76.var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.76-var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.pid -S /var/run/gluster/b6976f9ddc3abc90e1dfa91c35a7c611.socket --brick-name /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_54c06cdf52c6fe1e3a3f8a9d39cde33d/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.log --xlator-option *-posix.glusterd-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ5492 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/ec468453b11caed5259f5e1abcc65b44.socket --xlator-option *replicate*.node-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94
>
>Jun 08 07:25:17 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 07:25:20 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 08:11:14 Periodic health check status: node d942fe6c0ee5691b7cc263968f97b650 up=true
>[heketi] INFO 2018/06/08 08:11:14 Cleaned 0 nodes from health cache
>[negroni] Started POST /volumes
>[heketi] INFO 2018/06/08 08:12:58 Allocating brick set #0
>[negroni] Completed 202 Accepted in 12.839476ms
>[asynchttp] INFO 2018/06/08 08:12:58 asynchttp.go:288: Started job 3424e17d02fffd5a382569865106ef7e
>[heketi] INFO 2018/06/08 08:12:58 Started async operation: Create Volume
>[negroni] Started GET /queue/3424e17d02fffd5a382569865106ef7e
>[negroni] Completed 200 OK in 120.467Âµs
>[heketi] INFO 2018/06/08 08:12:58 Creating brick 0e79afb5f11fb53b2e0ca8f673c07abb
>[heketi] INFO 2018/06/08 08:12:58 Creating brick 3fc4d1412dc05e2bffb77dbc6fe5da99
>[heketi] INFO 2018/06/08 08:12:58 Creating brick ec22e3d2714797080dd7b0580ee72839
>[kubeexec] DEBUG 2018/06/08 08:12:58 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir -p /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_0e79afb5f11fb53b2e0ca8f673c07abb
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:12:58 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir -p /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_3fc4d1412dc05e2bffb77dbc6fe5da99
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:12:58 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir -p /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_ec22e3d2714797080dd7b0580ee72839
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:12:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvcreate --autobackup=n --poolmetadatasize 53248K --chunksize 256K --size 10485760K --thin vg_b861af9cc67f5bf8d5a8f9e38f56b14e/tp_0e79afb5f11fb53b2e0ca8f673c07abb --virtualsize 10485760K --name brick_0e79afb5f11fb53b2e0ca8f673c07abb
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_0e79afb5f11fb53b2e0ca8f673c07abb" created.
>[kubeexec] DEBUG 2018/06/08 08:12:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvcreate --autobackup=n --poolmetadatasize 53248K --chunksize 256K --size 10485760K --thin vg_9394bc70699b006c5460c9f654cf345f/tp_3fc4d1412dc05e2bffb77dbc6fe5da99 --virtualsize 10485760K --name brick_3fc4d1412dc05e2bffb77dbc6fe5da99
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_3fc4d1412dc05e2bffb77dbc6fe5da99" created.
>[kubeexec] DEBUG 2018/06/08 08:12:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvcreate --autobackup=n --poolmetadatasize 53248K --chunksize 256K --size 10485760K --thin vg_3a4297677881963e3f80124971d50eea/tp_ec22e3d2714797080dd7b0580ee72839 --virtualsize 10485760K --name brick_ec22e3d2714797080dd7b0580ee72839
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_ec22e3d2714797080dd7b0580ee72839" created.
>[kubeexec] DEBUG 2018/06/08 08:12:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_0e79afb5f11fb53b2e0ca8f673c07abb
>Result: meta-data=/dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_0e79afb5f11fb53b2e0ca8f673c07abb isize=512    agcount=16, agsize=163840 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=2621440, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:12:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_3fc4d1412dc05e2bffb77dbc6fe5da99
>Result: meta-data=/dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_3fc4d1412dc05e2bffb77dbc6fe5da99 isize=512    agcount=16, agsize=163840 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=2621440, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:12:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_ec22e3d2714797080dd7b0580ee72839
>Result: meta-data=/dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_ec22e3d2714797080dd7b0580ee72839 isize=512    agcount=16, agsize=163840 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=2621440, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:12:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: awk "BEGIN {print \"/dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_0e79afb5f11fb53b2e0ca8f673c07abb /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_0e79afb5f11fb53b2e0ca8f673c07abb xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:12:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: awk "BEGIN {print \"/dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_3fc4d1412dc05e2bffb77dbc6fe5da99 /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_3fc4d1412dc05e2bffb77dbc6fe5da99 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:12:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: awk "BEGIN {print \"/dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_ec22e3d2714797080dd7b0580ee72839 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_ec22e3d2714797080dd7b0580ee72839 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:12:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_0e79afb5f11fb53b2e0ca8f673c07abb /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_0e79afb5f11fb53b2e0ca8f673c07abb
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:12:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_ec22e3d2714797080dd7b0580ee72839 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_ec22e3d2714797080dd7b0580ee72839
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:12:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_3fc4d1412dc05e2bffb77dbc6fe5da99 /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_3fc4d1412dc05e2bffb77dbc6fe5da99
>Result: 
>[negroni] Started GET /queue/3424e17d02fffd5a382569865106ef7e
>[negroni] Completed 200 OK in 119.959Âµs
>[kubeexec] DEBUG 2018/06/08 08:12:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_0e79afb5f11fb53b2e0ca8f673c07abb/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:12:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_ec22e3d2714797080dd7b0580ee72839/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:12:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_3fc4d1412dc05e2bffb77dbc6fe5da99/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:12:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: chown :2000 /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_0e79afb5f11fb53b2e0ca8f673c07abb/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:12:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: chown :2000 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_ec22e3d2714797080dd7b0580ee72839/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:12:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: chown :2000 /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_3fc4d1412dc05e2bffb77dbc6fe5da99/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:13:00 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: chmod 2775 /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_0e79afb5f11fb53b2e0ca8f673c07abb/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:13:00 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: chmod 2775 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_ec22e3d2714797080dd7b0580ee72839/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:13:00 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: chmod 2775 /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_3fc4d1412dc05e2bffb77dbc6fe5da99/brick
>Result: 
>[cmdexec] INFO 2018/06/08 08:13:00 Creating volume cns-vol_glusterfs_mongodb5_c14e78ae-6af3-11e8-ab19-005056a5f18a replica 3
>[kubeexec] DEBUG 2018/06/08 08:13:00 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: gluster --mode=script volume create cns-vol_glusterfs_mongodb5_c14e78ae-6af3-11e8-ab19-005056a5f18a replica 3 10.70.46.187:/var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_0e79afb5f11fb53b2e0ca8f673c07abb/brick 10.70.46.122:/var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_ec22e3d2714797080dd7b0580ee72839/brick 10.70.47.76:/var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_3fc4d1412dc05e2bffb77dbc6fe5da99/brick
>Result: volume create: cns-vol_glusterfs_mongodb5_c14e78ae-6af3-11e8-ab19-005056a5f18a: success: please start the volume to access data
>[negroni] Started GET /queue/3424e17d02fffd5a382569865106ef7e
>[negroni] Completed 200 OK in 119.53Âµs
>[negroni] Started GET /queue/3424e17d02fffd5a382569865106ef7e
>[negroni] Completed 200 OK in 117.531Âµs
>[kubeexec] DEBUG 2018/06/08 08:13:02 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: gluster --mode=script volume start cns-vol_glusterfs_mongodb5_c14e78ae-6af3-11e8-ab19-005056a5f18a
>Result: volume start: cns-vol_glusterfs_mongodb5_c14e78ae-6af3-11e8-ab19-005056a5f18a: success
>[heketi] INFO 2018/06/08 08:13:02 Create Volume succeeded
>[asynchttp] INFO 2018/06/08 08:13:02 asynchttp.go:292: Completed job 3424e17d02fffd5a382569865106ef7e in 3.784685973s
>[negroni] Started GET /queue/3424e17d02fffd5a382569865106ef7e
>[negroni] Completed 303 See Other in 120.891Âµs
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 2.188195ms
>[negroni] Started GET /clusters/0a73c60efdd4673113b668afea101e6d
>[negroni] Completed 200 OK in 215.673Âµs
>[negroni] Started GET /nodes/278bd6b4e16a8e62ef15aaae22e6abc1
>[negroni] Completed 200 OK in 3.4443ms
>[negroni] Started GET /nodes/70423cc0bcd044fe5ba8bbbd256a3e49
>[negroni] Completed 200 OK in 710.118Âµs
>[negroni] Started GET /nodes/d942fe6c0ee5691b7cc263968f97b650
>[negroni] Completed 200 OK in 625.354Âµs
>[heketi] INFO 2018/06/08 08:13:14 Starting Node Health Status refresh
>[cmdexec] INFO 2018/06/08 08:13:14 Check Glusterd service status in node dhcp46-187.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 08:13:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:33 UTC; 1h 49min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48fed567_6ae4_11e8_bc4b_005056a5f18a.slice/docker-a506d9aaca4b440d184df40d01e90b2040811c8090bf4c19d892081307b28c0c.scope/system.slice/glusterd.service
>           ââ 433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ1228 /usr/sbin/glusterfsd -s 10.70.46.187 --volfile-id heketidbstorage.10.70.46.187.var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.187-var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.pid -S /var/run/gluster/c08898c2e3b0190c62df3a9d4a013c92.socket --brick-name /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.log --xlator-option *-posix.glusterd-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ7786 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/8fb729a0bbd51b204491c603c84acaa4.socket --xlator-option *replicate*.node-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034
>
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Unit glusterd.service entered failed state.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: glusterd.service failed.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:33 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 08:13:14 Periodic health check status: node 278bd6b4e16a8e62ef15aaae22e6abc1 up=true
>[cmdexec] INFO 2018/06/08 08:13:14 Check Glusterd service status in node dhcp46-122.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 08:13:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:42 UTC; 1h 49min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4900c968_6ae4_11e8_bc4b_005056a5f18a.slice/docker-438fb5fb16a82a3e701b28fa1f1ef8623dc37ca06374c604e5781a2b2ab2e06d.scope/system.slice/glusterd.service
>           ââ 433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ1106 /usr/sbin/glusterfsd -s 10.70.46.122 --volfile-id heketidbstorage.10.70.46.122.var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.122-var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.pid -S /var/run/gluster/9fa3ac3b12826912287adc88c26ac134.socket --brick-name /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_29e5b36d921cca1b0699871d9727c932/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.log --xlator-option *-posix.glusterd-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ7491 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f8994cdb09351d83e70da08261d99c64.socket --xlator-option *replicate*.node-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0
>
>Jun 08 06:23:40 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:42 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 08:13:14 Periodic health check status: node 70423cc0bcd044fe5ba8bbbd256a3e49 up=true
>[cmdexec] INFO 2018/06/08 08:13:14 Check Glusterd service status in node dhcp47-76.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 08:13:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 07:25:20 UTC; 47min ago
>  Process: 828 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 830 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5cb2299_6aec_11e8_bc4b_005056a5f18a.slice/docker-7f9223abc9a8641cea68f374f4039235515978ee2dab970ac6cbef182c535426.scope/system.slice/glusterd.service
>           ââ 830 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ1197 /usr/sbin/glusterfsd -s 10.70.47.76 --volfile-id heketidbstorage.10.70.47.76.var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.76-var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.pid -S /var/run/gluster/b6976f9ddc3abc90e1dfa91c35a7c611.socket --brick-name /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_54c06cdf52c6fe1e3a3f8a9d39cde33d/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.log --xlator-option *-posix.glusterd-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ5842 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/ec468453b11caed5259f5e1abcc65b44.socket --xlator-option *replicate*.node-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94
>
>Jun 08 07:25:17 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 07:25:20 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 08:13:14 Periodic health check status: node d942fe6c0ee5691b7cc263968f97b650 up=true
>[heketi] INFO 2018/06/08 08:13:14 Cleaned 0 nodes from health cache
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 170.677Âµs
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 659.525Âµs
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 620.787Âµs
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 1.104315ms
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 468.786Âµs
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 787.568Âµs
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 1.227488ms
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 707.572Âµs
>[negroni] Started DELETE /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 202 Accepted in 8.0761ms
>[asynchttp] INFO 2018/06/08 08:13:35 asynchttp.go:288: Started job 03238a47b86fc10f8b57b281e7ab0514
>[heketi] INFO 2018/06/08 08:13:35 Started async operation: Delete Volume
>[negroni] Started GET /queue/03238a47b86fc10f8b57b281e7ab0514
>[negroni] Completed 200 OK in 98.323Âµs
>[kubeexec] DEBUG 2018/06/08 08:13:35 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script snapshot list vol_9c7d5f0f473cce6914804135f0b8ddcd --xml
>Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
><cliOutput>
>  <opRet>-1</opRet>
>  <opErrno>30806</opErrno>
>  <opErrstr>Volume (vol_9c7d5f0f473cce6914804135f0b8ddcd) does not exist</opErrstr>
></cliOutput>
>[kubeexec] ERROR 2018/06/08 08:13:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume stop vol_9c7d5f0f473cce6914804135f0b8ddcd force] on glusterfs-storage-gxp7c: Err[command terminated with exit code 1]: Stdout []: Stderr [volume stop: vol_9c7d5f0f473cce6914804135f0b8ddcd: failed: Volume vol_9c7d5f0f473cce6914804135f0b8ddcd does not exist
>]
>[cmdexec] ERROR 2018/06/08 08:13:36 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:144: Unable to stop volume vol_9c7d5f0f473cce6914804135f0b8ddcd: Unable to execute command on glusterfs-storage-gxp7c: volume stop: vol_9c7d5f0f473cce6914804135f0b8ddcd: failed: Volume vol_9c7d5f0f473cce6914804135f0b8ddcd does not exist
>[kubeexec] ERROR 2018/06/08 08:13:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume delete vol_9c7d5f0f473cce6914804135f0b8ddcd] on glusterfs-storage-gxp7c: Err[command terminated with exit code 1]: Stdout []: Stderr [volume delete: vol_9c7d5f0f473cce6914804135f0b8ddcd: failed: Volume vol_9c7d5f0f473cce6914804135f0b8ddcd does not exist
>]
>[cmdexec] ERROR 2018/06/08 08:13:36 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:153: Unable to delete volume vol_9c7d5f0f473cce6914804135f0b8ddcd: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c7d5f0f473cce6914804135f0b8ddcd: failed: Volume vol_9c7d5f0f473cce6914804135f0b8ddcd does not exist
>[heketi] ERROR 2018/06/08 08:13:36 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:525: Unable to delete volume: Unable to delete volume vol_9c7d5f0f473cce6914804135f0b8ddcd: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c7d5f0f473cce6914804135f0b8ddcd: failed: Volume vol_9c7d5f0f473cce6914804135f0b8ddcd does not exist
>[heketi] ERROR 2018/06/08 08:13:36 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:429: Error executing delete volume: Unable to delete volume vol_9c7d5f0f473cce6914804135f0b8ddcd: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c7d5f0f473cce6914804135f0b8ddcd: failed: Volume vol_9c7d5f0f473cce6914804135f0b8ddcd does not exist
>[heketi] ERROR 2018/06/08 08:13:36 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:1185: Delete Volume Failed: Unable to delete volume vol_9c7d5f0f473cce6914804135f0b8ddcd: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c7d5f0f473cce6914804135f0b8ddcd: failed: Volume vol_9c7d5f0f473cce6914804135f0b8ddcd does not exist
>[asynchttp] INFO 2018/06/08 08:13:36 asynchttp.go:292: Completed job 03238a47b86fc10f8b57b281e7ab0514 in 650.80344ms
>[negroni] Started GET /queue/03238a47b86fc10f8b57b281e7ab0514
>[negroni] Completed 500 Internal Server Error in 183.468Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 2.081463ms
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 1.576195ms
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 1.114368ms
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 1.178338ms
>[heketi] INFO 2018/06/08 08:15:14 Starting Node Health Status refresh
>[cmdexec] INFO 2018/06/08 08:15:14 Check Glusterd service status in node dhcp46-187.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 08:15:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:33 UTC; 1h 51min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48fed567_6ae4_11e8_bc4b_005056a5f18a.slice/docker-a506d9aaca4b440d184df40d01e90b2040811c8090bf4c19d892081307b28c0c.scope/system.slice/glusterd.service
>           ââ 433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ1228 /usr/sbin/glusterfsd -s 10.70.46.187 --volfile-id heketidbstorage.10.70.46.187.var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.187-var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.pid -S /var/run/gluster/c08898c2e3b0190c62df3a9d4a013c92.socket --brick-name /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.log --xlator-option *-posix.glusterd-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ7786 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/8fb729a0bbd51b204491c603c84acaa4.socket --xlator-option *replicate*.node-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034
>
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Unit glusterd.service entered failed state.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: glusterd.service failed.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:33 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 08:15:14 Periodic health check status: node 278bd6b4e16a8e62ef15aaae22e6abc1 up=true
>[cmdexec] INFO 2018/06/08 08:15:14 Check Glusterd service status in node dhcp46-122.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 08:15:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:42 UTC; 1h 51min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4900c968_6ae4_11e8_bc4b_005056a5f18a.slice/docker-438fb5fb16a82a3e701b28fa1f1ef8623dc37ca06374c604e5781a2b2ab2e06d.scope/system.slice/glusterd.service
>           ââ 433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ1106 /usr/sbin/glusterfsd -s 10.70.46.122 --volfile-id heketidbstorage.10.70.46.122.var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.122-var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.pid -S /var/run/gluster/9fa3ac3b12826912287adc88c26ac134.socket --brick-name /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_29e5b36d921cca1b0699871d9727c932/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.log --xlator-option *-posix.glusterd-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ7491 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f8994cdb09351d83e70da08261d99c64.socket --xlator-option *replicate*.node-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0
>
>Jun 08 06:23:40 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:42 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 08:15:14 Periodic health check status: node 70423cc0bcd044fe5ba8bbbd256a3e49 up=true
>[cmdexec] INFO 2018/06/08 08:15:14 Check Glusterd service status in node dhcp47-76.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 08:15:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 07:25:20 UTC; 49min ago
>  Process: 828 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 830 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5cb2299_6aec_11e8_bc4b_005056a5f18a.slice/docker-7f9223abc9a8641cea68f374f4039235515978ee2dab970ac6cbef182c535426.scope/system.slice/glusterd.service
>           ââ 830 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ1197 /usr/sbin/glusterfsd -s 10.70.47.76 --volfile-id heketidbstorage.10.70.47.76.var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.76-var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.pid -S /var/run/gluster/b6976f9ddc3abc90e1dfa91c35a7c611.socket --brick-name /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_54c06cdf52c6fe1e3a3f8a9d39cde33d/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.log --xlator-option *-posix.glusterd-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ5842 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/ec468453b11caed5259f5e1abcc65b44.socket --xlator-option *replicate*.node-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94
>
>Jun 08 07:25:17 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 07:25:20 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 08:15:14 Periodic health check status: node d942fe6c0ee5691b7cc263968f97b650 up=true
>[heketi] INFO 2018/06/08 08:15:14 Cleaned 0 nodes from health cache
>[heketi] INFO 2018/06/08 08:17:14 Starting Node Health Status refresh
>[cmdexec] INFO 2018/06/08 08:17:14 Check Glusterd service status in node dhcp46-187.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 08:17:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:33 UTC; 1h 53min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48fed567_6ae4_11e8_bc4b_005056a5f18a.slice/docker-a506d9aaca4b440d184df40d01e90b2040811c8090bf4c19d892081307b28c0c.scope/system.slice/glusterd.service
>           ââ 433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ1228 /usr/sbin/glusterfsd -s 10.70.46.187 --volfile-id heketidbstorage.10.70.46.187.var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.187-var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.pid -S /var/run/gluster/c08898c2e3b0190c62df3a9d4a013c92.socket --brick-name /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.log --xlator-option *-posix.glusterd-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ7786 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/8fb729a0bbd51b204491c603c84acaa4.socket --xlator-option *replicate*.node-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034
>
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Unit glusterd.service entered failed state.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: glusterd.service failed.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:33 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 08:17:14 Periodic health check status: node 278bd6b4e16a8e62ef15aaae22e6abc1 up=true
>[cmdexec] INFO 2018/06/08 08:17:14 Check Glusterd service status in node dhcp46-122.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 08:17:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:42 UTC; 1h 53min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4900c968_6ae4_11e8_bc4b_005056a5f18a.slice/docker-438fb5fb16a82a3e701b28fa1f1ef8623dc37ca06374c604e5781a2b2ab2e06d.scope/system.slice/glusterd.service
>           ââ 433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ1106 /usr/sbin/glusterfsd -s 10.70.46.122 --volfile-id heketidbstorage.10.70.46.122.var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.122-var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.pid -S /var/run/gluster/9fa3ac3b12826912287adc88c26ac134.socket --brick-name /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_29e5b36d921cca1b0699871d9727c932/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.log --xlator-option *-posix.glusterd-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ7491 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f8994cdb09351d83e70da08261d99c64.socket --xlator-option *replicate*.node-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0
>
>Jun 08 06:23:40 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:42 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 08:17:14 Periodic health check status: node 70423cc0bcd044fe5ba8bbbd256a3e49 up=true
>[cmdexec] INFO 2018/06/08 08:17:14 Check Glusterd service status in node dhcp47-76.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 08:17:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 07:25:20 UTC; 51min ago
>  Process: 828 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 830 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5cb2299_6aec_11e8_bc4b_005056a5f18a.slice/docker-7f9223abc9a8641cea68f374f4039235515978ee2dab970ac6cbef182c535426.scope/system.slice/glusterd.service
>           ââ 830 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ1197 /usr/sbin/glusterfsd -s 10.70.47.76 --volfile-id heketidbstorage.10.70.47.76.var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.76-var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.pid -S /var/run/gluster/b6976f9ddc3abc90e1dfa91c35a7c611.socket --brick-name /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_54c06cdf52c6fe1e3a3f8a9d39cde33d/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.log --xlator-option *-posix.glusterd-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ5842 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/ec468453b11caed5259f5e1abcc65b44.socket --xlator-option *replicate*.node-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94
>
>Jun 08 07:25:17 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 07:25:20 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 08:17:14 Periodic health check status: node d942fe6c0ee5691b7cc263968f97b650 up=true
>[heketi] INFO 2018/06/08 08:17:14 Cleaned 0 nodes from health cache
>[heketi] INFO 2018/06/08 08:19:14 Starting Node Health Status refresh
>[cmdexec] INFO 2018/06/08 08:19:14 Check Glusterd service status in node dhcp46-187.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 08:19:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:33 UTC; 1h 55min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48fed567_6ae4_11e8_bc4b_005056a5f18a.slice/docker-a506d9aaca4b440d184df40d01e90b2040811c8090bf4c19d892081307b28c0c.scope/system.slice/glusterd.service
>           ââ 433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ1228 /usr/sbin/glusterfsd -s 10.70.46.187 --volfile-id heketidbstorage.10.70.46.187.var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.187-var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.pid -S /var/run/gluster/c08898c2e3b0190c62df3a9d4a013c92.socket --brick-name /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.log --xlator-option *-posix.glusterd-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ7786 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/8fb729a0bbd51b204491c603c84acaa4.socket --xlator-option *replicate*.node-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034
>
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Unit glusterd.service entered failed state.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: glusterd.service failed.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:33 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 08:19:14 Periodic health check status: node 278bd6b4e16a8e62ef15aaae22e6abc1 up=true
>[cmdexec] INFO 2018/06/08 08:19:14 Check Glusterd service status in node dhcp46-122.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 08:19:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:42 UTC; 1h 55min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4900c968_6ae4_11e8_bc4b_005056a5f18a.slice/docker-438fb5fb16a82a3e701b28fa1f1ef8623dc37ca06374c604e5781a2b2ab2e06d.scope/system.slice/glusterd.service
>           ââ 433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ1106 /usr/sbin/glusterfsd -s 10.70.46.122 --volfile-id heketidbstorage.10.70.46.122.var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.122-var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.pid -S /var/run/gluster/9fa3ac3b12826912287adc88c26ac134.socket --brick-name /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_29e5b36d921cca1b0699871d9727c932/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.log --xlator-option *-posix.glusterd-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ7491 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f8994cdb09351d83e70da08261d99c64.socket --xlator-option *replicate*.node-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0
>
>Jun 08 06:23:40 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:42 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 08:19:14 Periodic health check status: node 70423cc0bcd044fe5ba8bbbd256a3e49 up=true
>[cmdexec] INFO 2018/06/08 08:19:14 Check Glusterd service status in node dhcp47-76.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 08:19:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 07:25:20 UTC; 53min ago
>  Process: 828 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 830 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5cb2299_6aec_11e8_bc4b_005056a5f18a.slice/docker-7f9223abc9a8641cea68f374f4039235515978ee2dab970ac6cbef182c535426.scope/system.slice/glusterd.service
>           ââ 830 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ1197 /usr/sbin/glusterfsd -s 10.70.47.76 --volfile-id heketidbstorage.10.70.47.76.var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.76-var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.pid -S /var/run/gluster/b6976f9ddc3abc90e1dfa91c35a7c611.socket --brick-name /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_54c06cdf52c6fe1e3a3f8a9d39cde33d/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.log --xlator-option *-posix.glusterd-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ5842 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/ec468453b11caed5259f5e1abcc65b44.socket --xlator-option *replicate*.node-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94
>
>Jun 08 07:25:17 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 07:25:20 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 08:19:14 Periodic health check status: node d942fe6c0ee5691b7cc263968f97b650 up=true
>[heketi] INFO 2018/06/08 08:19:14 Cleaned 0 nodes from health cache
>[heketi] INFO 2018/06/08 08:21:14 Starting Node Health Status refresh
>[cmdexec] INFO 2018/06/08 08:21:14 Check Glusterd service status in node dhcp46-187.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 08:21:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:33 UTC; 1h 57min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48fed567_6ae4_11e8_bc4b_005056a5f18a.slice/docker-a506d9aaca4b440d184df40d01e90b2040811c8090bf4c19d892081307b28c0c.scope/system.slice/glusterd.service
>           ââ 433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ1228 /usr/sbin/glusterfsd -s 10.70.46.187 --volfile-id heketidbstorage.10.70.46.187.var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.187-var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.pid -S /var/run/gluster/c08898c2e3b0190c62df3a9d4a013c92.socket --brick-name /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.log --xlator-option *-posix.glusterd-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ7786 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/8fb729a0bbd51b204491c603c84acaa4.socket --xlator-option *replicate*.node-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034
>
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Unit glusterd.service entered failed state.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: glusterd.service failed.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:33 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 08:21:14 Periodic health check status: node 278bd6b4e16a8e62ef15aaae22e6abc1 up=true
>[cmdexec] INFO 2018/06/08 08:21:14 Check Glusterd service status in node dhcp46-122.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 08:21:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:42 UTC; 1h 57min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4900c968_6ae4_11e8_bc4b_005056a5f18a.slice/docker-438fb5fb16a82a3e701b28fa1f1ef8623dc37ca06374c604e5781a2b2ab2e06d.scope/system.slice/glusterd.service
>           ââ 433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ1106 /usr/sbin/glusterfsd -s 10.70.46.122 --volfile-id heketidbstorage.10.70.46.122.var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.122-var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.pid -S /var/run/gluster/9fa3ac3b12826912287adc88c26ac134.socket --brick-name /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_29e5b36d921cca1b0699871d9727c932/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.log --xlator-option *-posix.glusterd-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ7491 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f8994cdb09351d83e70da08261d99c64.socket --xlator-option *replicate*.node-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0
>
>Jun 08 06:23:40 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:42 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 08:21:14 Periodic health check status: node 70423cc0bcd044fe5ba8bbbd256a3e49 up=true
>[cmdexec] INFO 2018/06/08 08:21:14 Check Glusterd service status in node dhcp47-76.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 08:21:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 07:25:20 UTC; 55min ago
>  Process: 828 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 830 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5cb2299_6aec_11e8_bc4b_005056a5f18a.slice/docker-7f9223abc9a8641cea68f374f4039235515978ee2dab970ac6cbef182c535426.scope/system.slice/glusterd.service
>           ââ 830 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ1197 /usr/sbin/glusterfsd -s 10.70.47.76 --volfile-id heketidbstorage.10.70.47.76.var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.76-var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.pid -S /var/run/gluster/b6976f9ddc3abc90e1dfa91c35a7c611.socket --brick-name /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_54c06cdf52c6fe1e3a3f8a9d39cde33d/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.log --xlator-option *-posix.glusterd-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ5842 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/ec468453b11caed5259f5e1abcc65b44.socket --xlator-option *replicate*.node-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94
>
>Jun 08 07:25:17 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 07:25:20 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 08:21:14 Periodic health check status: node d942fe6c0ee5691b7cc263968f97b650 up=true
>[heketi] INFO 2018/06/08 08:21:14 Cleaned 0 nodes from health cache
>[heketi] INFO 2018/06/08 08:23:14 Starting Node Health Status refresh
>[cmdexec] INFO 2018/06/08 08:23:14 Check Glusterd service status in node dhcp46-187.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 08:23:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:33 UTC; 1h 59min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48fed567_6ae4_11e8_bc4b_005056a5f18a.slice/docker-a506d9aaca4b440d184df40d01e90b2040811c8090bf4c19d892081307b28c0c.scope/system.slice/glusterd.service
>           ââ 433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ1228 /usr/sbin/glusterfsd -s 10.70.46.187 --volfile-id heketidbstorage.10.70.46.187.var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.187-var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.pid -S /var/run/gluster/c08898c2e3b0190c62df3a9d4a013c92.socket --brick-name /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.log --xlator-option *-posix.glusterd-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ7786 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/8fb729a0bbd51b204491c603c84acaa4.socket --xlator-option *replicate*.node-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034
>
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Unit glusterd.service entered failed state.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: glusterd.service failed.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:33 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 08:23:14 Periodic health check status: node 278bd6b4e16a8e62ef15aaae22e6abc1 up=true
>[cmdexec] INFO 2018/06/08 08:23:14 Check Glusterd service status in node dhcp46-122.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 08:23:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:42 UTC; 1h 59min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4900c968_6ae4_11e8_bc4b_005056a5f18a.slice/docker-438fb5fb16a82a3e701b28fa1f1ef8623dc37ca06374c604e5781a2b2ab2e06d.scope/system.slice/glusterd.service
>           ââ 433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ1106 /usr/sbin/glusterfsd -s 10.70.46.122 --volfile-id heketidbstorage.10.70.46.122.var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.122-var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.pid -S /var/run/gluster/9fa3ac3b12826912287adc88c26ac134.socket --brick-name /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_29e5b36d921cca1b0699871d9727c932/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.log --xlator-option *-posix.glusterd-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ7491 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f8994cdb09351d83e70da08261d99c64.socket --xlator-option *replicate*.node-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0
>
>Jun 08 06:23:40 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:42 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 08:23:14 Periodic health check status: node 70423cc0bcd044fe5ba8bbbd256a3e49 up=true
>[cmdexec] INFO 2018/06/08 08:23:14 Check Glusterd service status in node dhcp47-76.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 08:23:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 07:25:20 UTC; 57min ago
>  Process: 828 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 830 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5cb2299_6aec_11e8_bc4b_005056a5f18a.slice/docker-7f9223abc9a8641cea68f374f4039235515978ee2dab970ac6cbef182c535426.scope/system.slice/glusterd.service
>           ââ 830 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ1197 /usr/sbin/glusterfsd -s 10.70.47.76 --volfile-id heketidbstorage.10.70.47.76.var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.76-var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.pid -S /var/run/gluster/b6976f9ddc3abc90e1dfa91c35a7c611.socket --brick-name /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_54c06cdf52c6fe1e3a3f8a9d39cde33d/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.log --xlator-option *-posix.glusterd-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ5842 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/ec468453b11caed5259f5e1abcc65b44.socket --xlator-option *replicate*.node-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94
>
>Jun 08 07:25:17 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 07:25:20 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 08:23:14 Periodic health check status: node d942fe6c0ee5691b7cc263968f97b650 up=true
>[heketi] INFO 2018/06/08 08:23:14 Cleaned 0 nodes from health cache
>[heketi] INFO 2018/06/08 08:25:14 Starting Node Health Status refresh
>[cmdexec] INFO 2018/06/08 08:25:14 Check Glusterd service status in node dhcp46-187.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 08:25:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:33 UTC; 2h 1min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48fed567_6ae4_11e8_bc4b_005056a5f18a.slice/docker-a506d9aaca4b440d184df40d01e90b2040811c8090bf4c19d892081307b28c0c.scope/system.slice/glusterd.service
>           ââ 433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ1228 /usr/sbin/glusterfsd -s 10.70.46.187 --volfile-id heketidbstorage.10.70.46.187.var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.187-var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.pid -S /var/run/gluster/c08898c2e3b0190c62df3a9d4a013c92.socket --brick-name /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.log --xlator-option *-posix.glusterd-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ7786 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/8fb729a0bbd51b204491c603c84acaa4.socket --xlator-option *replicate*.node-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034
>
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Unit glusterd.service entered failed state.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: glusterd.service failed.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:33 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 08:25:14 Periodic health check status: node 278bd6b4e16a8e62ef15aaae22e6abc1 up=true
>[cmdexec] INFO 2018/06/08 08:25:14 Check Glusterd service status in node dhcp46-122.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 08:25:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:42 UTC; 2h 1min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4900c968_6ae4_11e8_bc4b_005056a5f18a.slice/docker-438fb5fb16a82a3e701b28fa1f1ef8623dc37ca06374c604e5781a2b2ab2e06d.scope/system.slice/glusterd.service
>           ââ 433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ1106 /usr/sbin/glusterfsd -s 10.70.46.122 --volfile-id heketidbstorage.10.70.46.122.var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.122-var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.pid -S /var/run/gluster/9fa3ac3b12826912287adc88c26ac134.socket --brick-name /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_29e5b36d921cca1b0699871d9727c932/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.log --xlator-option *-posix.glusterd-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ7491 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f8994cdb09351d83e70da08261d99c64.socket --xlator-option *replicate*.node-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0
>
>Jun 08 06:23:40 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:42 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 08:25:14 Periodic health check status: node 70423cc0bcd044fe5ba8bbbd256a3e49 up=true
>[cmdexec] INFO 2018/06/08 08:25:14 Check Glusterd service status in node dhcp47-76.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 08:25:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 07:25:20 UTC; 59min ago
>  Process: 828 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 830 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5cb2299_6aec_11e8_bc4b_005056a5f18a.slice/docker-7f9223abc9a8641cea68f374f4039235515978ee2dab970ac6cbef182c535426.scope/system.slice/glusterd.service
>           ââ 830 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ1197 /usr/sbin/glusterfsd -s 10.70.47.76 --volfile-id heketidbstorage.10.70.47.76.var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.76-var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.pid -S /var/run/gluster/b6976f9ddc3abc90e1dfa91c35a7c611.socket --brick-name /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_54c06cdf52c6fe1e3a3f8a9d39cde33d/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.log --xlator-option *-posix.glusterd-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ5842 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/ec468453b11caed5259f5e1abcc65b44.socket --xlator-option *replicate*.node-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94
>
>Jun 08 07:25:17 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 07:25:20 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 08:25:14 Periodic health check status: node d942fe6c0ee5691b7cc263968f97b650 up=true
>[heketi] INFO 2018/06/08 08:25:14 Cleaned 0 nodes from health cache
>[heketi] INFO 2018/06/08 08:27:14 Starting Node Health Status refresh
>[cmdexec] INFO 2018/06/08 08:27:14 Check Glusterd service status in node dhcp46-187.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 08:27:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:33 UTC; 2h 3min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48fed567_6ae4_11e8_bc4b_005056a5f18a.slice/docker-a506d9aaca4b440d184df40d01e90b2040811c8090bf4c19d892081307b28c0c.scope/system.slice/glusterd.service
>           ââ 433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ1228 /usr/sbin/glusterfsd -s 10.70.46.187 --volfile-id heketidbstorage.10.70.46.187.var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.187-var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.pid -S /var/run/gluster/c08898c2e3b0190c62df3a9d4a013c92.socket --brick-name /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.log --xlator-option *-posix.glusterd-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ7786 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/8fb729a0bbd51b204491c603c84acaa4.socket --xlator-option *replicate*.node-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034
>
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Unit glusterd.service entered failed state.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: glusterd.service failed.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:33 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 08:27:14 Periodic health check status: node 278bd6b4e16a8e62ef15aaae22e6abc1 up=true
>[cmdexec] INFO 2018/06/08 08:27:14 Check Glusterd service status in node dhcp46-122.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 08:27:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:42 UTC; 2h 3min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4900c968_6ae4_11e8_bc4b_005056a5f18a.slice/docker-438fb5fb16a82a3e701b28fa1f1ef8623dc37ca06374c604e5781a2b2ab2e06d.scope/system.slice/glusterd.service
>           ââ 433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ1106 /usr/sbin/glusterfsd -s 10.70.46.122 --volfile-id heketidbstorage.10.70.46.122.var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.122-var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.pid -S /var/run/gluster/9fa3ac3b12826912287adc88c26ac134.socket --brick-name /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_29e5b36d921cca1b0699871d9727c932/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.log --xlator-option *-posix.glusterd-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ7491 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f8994cdb09351d83e70da08261d99c64.socket --xlator-option *replicate*.node-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0
>
>Jun 08 06:23:40 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:42 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 08:27:14 Periodic health check status: node 70423cc0bcd044fe5ba8bbbd256a3e49 up=true
>[cmdexec] INFO 2018/06/08 08:27:14 Check Glusterd service status in node dhcp47-76.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 08:27:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 07:25:20 UTC; 1h 1min ago
>  Process: 828 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 830 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5cb2299_6aec_11e8_bc4b_005056a5f18a.slice/docker-7f9223abc9a8641cea68f374f4039235515978ee2dab970ac6cbef182c535426.scope/system.slice/glusterd.service
>           ââ 830 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ1197 /usr/sbin/glusterfsd -s 10.70.47.76 --volfile-id heketidbstorage.10.70.47.76.var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.76-var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.pid -S /var/run/gluster/b6976f9ddc3abc90e1dfa91c35a7c611.socket --brick-name /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_54c06cdf52c6fe1e3a3f8a9d39cde33d/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.log --xlator-option *-posix.glusterd-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ5842 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/ec468453b11caed5259f5e1abcc65b44.socket --xlator-option *replicate*.node-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94
>
>Jun 08 07:25:17 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 07:25:20 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 08:27:14 Periodic health check status: node d942fe6c0ee5691b7cc263968f97b650 up=true
>[heketi] INFO 2018/06/08 08:27:14 Cleaned 0 nodes from health cache
>[heketi] INFO 2018/06/08 08:29:14 Starting Node Health Status refresh
>[cmdexec] INFO 2018/06/08 08:29:14 Check Glusterd service status in node dhcp46-187.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 08:29:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:33 UTC; 2h 5min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48fed567_6ae4_11e8_bc4b_005056a5f18a.slice/docker-a506d9aaca4b440d184df40d01e90b2040811c8090bf4c19d892081307b28c0c.scope/system.slice/glusterd.service
>           ââ 433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ1228 /usr/sbin/glusterfsd -s 10.70.46.187 --volfile-id heketidbstorage.10.70.46.187.var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.187-var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.pid -S /var/run/gluster/c08898c2e3b0190c62df3a9d4a013c92.socket --brick-name /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.log --xlator-option *-posix.glusterd-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ7786 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/8fb729a0bbd51b204491c603c84acaa4.socket --xlator-option *replicate*.node-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034
>
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Unit glusterd.service entered failed state.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: glusterd.service failed.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:33 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 08:29:14 Periodic health check status: node 278bd6b4e16a8e62ef15aaae22e6abc1 up=true
>[cmdexec] INFO 2018/06/08 08:29:14 Check Glusterd service status in node dhcp46-122.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 08:29:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:42 UTC; 2h 5min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4900c968_6ae4_11e8_bc4b_005056a5f18a.slice/docker-438fb5fb16a82a3e701b28fa1f1ef8623dc37ca06374c604e5781a2b2ab2e06d.scope/system.slice/glusterd.service
>           ââ 433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ1106 /usr/sbin/glusterfsd -s 10.70.46.122 --volfile-id heketidbstorage.10.70.46.122.var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.122-var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.pid -S /var/run/gluster/9fa3ac3b12826912287adc88c26ac134.socket --brick-name /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_29e5b36d921cca1b0699871d9727c932/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.log --xlator-option *-posix.glusterd-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ7491 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f8994cdb09351d83e70da08261d99c64.socket --xlator-option *replicate*.node-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0
>
>Jun 08 06:23:40 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:42 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 08:29:14 Periodic health check status: node 70423cc0bcd044fe5ba8bbbd256a3e49 up=true
>[cmdexec] INFO 2018/06/08 08:29:14 Check Glusterd service status in node dhcp47-76.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 08:29:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 07:25:20 UTC; 1h 3min ago
>  Process: 828 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 830 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5cb2299_6aec_11e8_bc4b_005056a5f18a.slice/docker-7f9223abc9a8641cea68f374f4039235515978ee2dab970ac6cbef182c535426.scope/system.slice/glusterd.service
>           ââ 830 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ1197 /usr/sbin/glusterfsd -s 10.70.47.76 --volfile-id heketidbstorage.10.70.47.76.var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.76-var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.pid -S /var/run/gluster/b6976f9ddc3abc90e1dfa91c35a7c611.socket --brick-name /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_54c06cdf52c6fe1e3a3f8a9d39cde33d/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.log --xlator-option *-posix.glusterd-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ5842 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/ec468453b11caed5259f5e1abcc65b44.socket --xlator-option *replicate*.node-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94
>
>Jun 08 07:25:17 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 07:25:20 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 08:29:14 Periodic health check status: node d942fe6c0ee5691b7cc263968f97b650 up=true
>[heketi] INFO 2018/06/08 08:29:14 Cleaned 0 nodes from health cache
>[heketi] INFO 2018/06/08 08:31:14 Starting Node Health Status refresh
>[cmdexec] INFO 2018/06/08 08:31:14 Check Glusterd service status in node dhcp46-187.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 08:31:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:33 UTC; 2h 7min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48fed567_6ae4_11e8_bc4b_005056a5f18a.slice/docker-a506d9aaca4b440d184df40d01e90b2040811c8090bf4c19d892081307b28c0c.scope/system.slice/glusterd.service
>           ââ 433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ1228 /usr/sbin/glusterfsd -s 10.70.46.187 --volfile-id heketidbstorage.10.70.46.187.var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.187-var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.pid -S /var/run/gluster/c08898c2e3b0190c62df3a9d4a013c92.socket --brick-name /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.log --xlator-option *-posix.glusterd-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ7786 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/8fb729a0bbd51b204491c603c84acaa4.socket --xlator-option *replicate*.node-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034
>
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Unit glusterd.service entered failed state.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: glusterd.service failed.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:33 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 08:31:14 Periodic health check status: node 278bd6b4e16a8e62ef15aaae22e6abc1 up=true
>[cmdexec] INFO 2018/06/08 08:31:14 Check Glusterd service status in node dhcp46-122.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 08:31:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:42 UTC; 2h 7min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4900c968_6ae4_11e8_bc4b_005056a5f18a.slice/docker-438fb5fb16a82a3e701b28fa1f1ef8623dc37ca06374c604e5781a2b2ab2e06d.scope/system.slice/glusterd.service
>           ââ 433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ1106 /usr/sbin/glusterfsd -s 10.70.46.122 --volfile-id heketidbstorage.10.70.46.122.var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.122-var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.pid -S /var/run/gluster/9fa3ac3b12826912287adc88c26ac134.socket --brick-name /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_29e5b36d921cca1b0699871d9727c932/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.log --xlator-option *-posix.glusterd-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ7491 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f8994cdb09351d83e70da08261d99c64.socket --xlator-option *replicate*.node-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0
>
>Jun 08 06:23:40 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:42 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 08:31:14 Periodic health check status: node 70423cc0bcd044fe5ba8bbbd256a3e49 up=true
>[cmdexec] INFO 2018/06/08 08:31:14 Check Glusterd service status in node dhcp47-76.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 08:31:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 07:25:20 UTC; 1h 5min ago
>  Process: 828 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 830 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5cb2299_6aec_11e8_bc4b_005056a5f18a.slice/docker-7f9223abc9a8641cea68f374f4039235515978ee2dab970ac6cbef182c535426.scope/system.slice/glusterd.service
>           ââ 830 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ1197 /usr/sbin/glusterfsd -s 10.70.47.76 --volfile-id heketidbstorage.10.70.47.76.var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.76-var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.pid -S /var/run/gluster/b6976f9ddc3abc90e1dfa91c35a7c611.socket --brick-name /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_54c06cdf52c6fe1e3a3f8a9d39cde33d/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.log --xlator-option *-posix.glusterd-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ5842 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/ec468453b11caed5259f5e1abcc65b44.socket --xlator-option *replicate*.node-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94
>
>Jun 08 07:25:17 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 07:25:20 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 08:31:14 Periodic health check status: node d942fe6c0ee5691b7cc263968f97b650 up=true
>[heketi] INFO 2018/06/08 08:31:14 Cleaned 0 nodes from health cache
>[heketi] INFO 2018/06/08 08:33:14 Starting Node Health Status refresh
>[cmdexec] INFO 2018/06/08 08:33:14 Check Glusterd service status in node dhcp46-187.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 08:33:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:33 UTC; 2h 9min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48fed567_6ae4_11e8_bc4b_005056a5f18a.slice/docker-a506d9aaca4b440d184df40d01e90b2040811c8090bf4c19d892081307b28c0c.scope/system.slice/glusterd.service
>           ââ 433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ1228 /usr/sbin/glusterfsd -s 10.70.46.187 --volfile-id heketidbstorage.10.70.46.187.var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.187-var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.pid -S /var/run/gluster/c08898c2e3b0190c62df3a9d4a013c92.socket --brick-name /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.log --xlator-option *-posix.glusterd-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ7786 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/8fb729a0bbd51b204491c603c84acaa4.socket --xlator-option *replicate*.node-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034
>
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Unit glusterd.service entered failed state.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: glusterd.service failed.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:33 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 08:33:14 Periodic health check status: node 278bd6b4e16a8e62ef15aaae22e6abc1 up=true
>[cmdexec] INFO 2018/06/08 08:33:14 Check Glusterd service status in node dhcp46-122.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 08:33:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:42 UTC; 2h 9min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4900c968_6ae4_11e8_bc4b_005056a5f18a.slice/docker-438fb5fb16a82a3e701b28fa1f1ef8623dc37ca06374c604e5781a2b2ab2e06d.scope/system.slice/glusterd.service
>           ââ 433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ1106 /usr/sbin/glusterfsd -s 10.70.46.122 --volfile-id heketidbstorage.10.70.46.122.var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.122-var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.pid -S /var/run/gluster/9fa3ac3b12826912287adc88c26ac134.socket --brick-name /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_29e5b36d921cca1b0699871d9727c932/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.log --xlator-option *-posix.glusterd-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ7491 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f8994cdb09351d83e70da08261d99c64.socket --xlator-option *replicate*.node-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0
>
>Jun 08 06:23:40 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:42 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 08:33:14 Periodic health check status: node 70423cc0bcd044fe5ba8bbbd256a3e49 up=true
>[cmdexec] INFO 2018/06/08 08:33:14 Check Glusterd service status in node dhcp47-76.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 08:33:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 07:25:20 UTC; 1h 7min ago
>  Process: 828 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 830 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5cb2299_6aec_11e8_bc4b_005056a5f18a.slice/docker-7f9223abc9a8641cea68f374f4039235515978ee2dab970ac6cbef182c535426.scope/system.slice/glusterd.service
>           ââ 830 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ1197 /usr/sbin/glusterfsd -s 10.70.47.76 --volfile-id heketidbstorage.10.70.47.76.var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.76-var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.pid -S /var/run/gluster/b6976f9ddc3abc90e1dfa91c35a7c611.socket --brick-name /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_54c06cdf52c6fe1e3a3f8a9d39cde33d/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.log --xlator-option *-posix.glusterd-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ5842 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/ec468453b11caed5259f5e1abcc65b44.socket --xlator-option *replicate*.node-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94
>
>Jun 08 07:25:17 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 07:25:20 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 08:33:14 Periodic health check status: node d942fe6c0ee5691b7cc263968f97b650 up=true
>[heketi] INFO 2018/06/08 08:33:14 Cleaned 0 nodes from health cache
>[negroni] Started POST /volumes
>[heketi] INFO 2018/06/08 08:33:33 Allocating brick set #0
>[negroni] Completed 202 Accepted in 14.35258ms
>[asynchttp] INFO 2018/06/08 08:33:33 asynchttp.go:288: Started job a6b906276aeeaaac96c8168db7c988bb
>[heketi] INFO 2018/06/08 08:33:33 Started async operation: Create Volume
>[heketi] INFO 2018/06/08 08:33:33 Creating brick ce0cbbac0ac897ad4fe0a2d0068195ae
>[negroni] Started GET /queue/a6b906276aeeaaac96c8168db7c988bb
>[negroni] Completed 200 OK in 129.321Âµs
>[heketi] INFO 2018/06/08 08:33:33 Creating brick 989fc70ef70fd18a678f7fc65ac3a56d
>[heketi] INFO 2018/06/08 08:33:33 Creating brick 53eb288b3c12a575d4cbef2bce0292ef
>[kubeexec] DEBUG 2018/06/08 08:33:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir -p /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_ce0cbbac0ac897ad4fe0a2d0068195ae
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:33:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir -p /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_53eb288b3c12a575d4cbef2bce0292ef
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:33:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir -p /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_989fc70ef70fd18a678f7fc65ac3a56d
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:33:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvcreate --autobackup=n --poolmetadatasize 8192K --chunksize 256K --size 1048576K --thin vg_d389f0278a774bd7443a09af960961d8/tp_53eb288b3c12a575d4cbef2bce0292ef --virtualsize 1048576K --name brick_53eb288b3c12a575d4cbef2bce0292ef
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_53eb288b3c12a575d4cbef2bce0292ef" created.
>[kubeexec] DEBUG 2018/06/08 08:33:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvcreate --autobackup=n --poolmetadatasize 8192K --chunksize 256K --size 1048576K --thin vg_3a4297677881963e3f80124971d50eea/tp_ce0cbbac0ac897ad4fe0a2d0068195ae --virtualsize 1048576K --name brick_ce0cbbac0ac897ad4fe0a2d0068195ae
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_ce0cbbac0ac897ad4fe0a2d0068195ae" created.
>[kubeexec] DEBUG 2018/06/08 08:33:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvcreate --autobackup=n --poolmetadatasize 8192K --chunksize 256K --size 1048576K --thin vg_9394bc70699b006c5460c9f654cf345f/tp_989fc70ef70fd18a678f7fc65ac3a56d --virtualsize 1048576K --name brick_989fc70ef70fd18a678f7fc65ac3a56d
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_989fc70ef70fd18a678f7fc65ac3a56d" created.
>[kubeexec] DEBUG 2018/06/08 08:33:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_53eb288b3c12a575d4cbef2bce0292ef
>Result: meta-data=/dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_53eb288b3c12a575d4cbef2bce0292ef isize=512    agcount=8, agsize=32768 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=262144, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:33:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_989fc70ef70fd18a678f7fc65ac3a56d
>Result: meta-data=/dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_989fc70ef70fd18a678f7fc65ac3a56d isize=512    agcount=8, agsize=32768 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=262144, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:33:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_ce0cbbac0ac897ad4fe0a2d0068195ae
>Result: meta-data=/dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_ce0cbbac0ac897ad4fe0a2d0068195ae isize=512    agcount=8, agsize=32768 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=262144, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:33:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: awk "BEGIN {print \"/dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_53eb288b3c12a575d4cbef2bce0292ef /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_53eb288b3c12a575d4cbef2bce0292ef xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:33:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: awk "BEGIN {print \"/dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_ce0cbbac0ac897ad4fe0a2d0068195ae /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_ce0cbbac0ac897ad4fe0a2d0068195ae xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:33:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: awk "BEGIN {print \"/dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_989fc70ef70fd18a678f7fc65ac3a56d /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_989fc70ef70fd18a678f7fc65ac3a56d xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:33:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_53eb288b3c12a575d4cbef2bce0292ef /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_53eb288b3c12a575d4cbef2bce0292ef
>Result: 
>[negroni] Started GET /queue/a6b906276aeeaaac96c8168db7c988bb
>[negroni] Completed 200 OK in 136.298Âµs
>[kubeexec] DEBUG 2018/06/08 08:33:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_989fc70ef70fd18a678f7fc65ac3a56d /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_989fc70ef70fd18a678f7fc65ac3a56d
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:33:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_ce0cbbac0ac897ad4fe0a2d0068195ae /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_ce0cbbac0ac897ad4fe0a2d0068195ae
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:33:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_53eb288b3c12a575d4cbef2bce0292ef/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:33:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_ce0cbbac0ac897ad4fe0a2d0068195ae/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:33:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_989fc70ef70fd18a678f7fc65ac3a56d/brick
>Result: 
>[cmdexec] INFO 2018/06/08 08:33:34 Creating volume vol_793f2fea1ecd5540c6c7b9011fef2fef replica 3
>[kubeexec] DEBUG 2018/06/08 08:33:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script volume create vol_793f2fea1ecd5540c6c7b9011fef2fef replica 3 10.70.46.122:/var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_ce0cbbac0ac897ad4fe0a2d0068195ae/brick 10.70.47.76:/var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_989fc70ef70fd18a678f7fc65ac3a56d/brick 10.70.46.187:/var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_53eb288b3c12a575d4cbef2bce0292ef/brick
>Result: volume create: vol_793f2fea1ecd5540c6c7b9011fef2fef: success: please start the volume to access data
>[negroni] Started GET /queue/a6b906276aeeaaac96c8168db7c988bb
>[negroni] Completed 200 OK in 228.369Âµs
>[negroni] Started GET /queue/a6b906276aeeaaac96c8168db7c988bb
>[negroni] Completed 200 OK in 177.846Âµs
>[negroni] Started GET /queue/a6b906276aeeaaac96c8168db7c988bb
>[negroni] Completed 200 OK in 165.067Âµs
>[negroni] Started GET /queue/a6b906276aeeaaac96c8168db7c988bb
>[negroni] Completed 200 OK in 245.969Âµs
>[kubeexec] DEBUG 2018/06/08 08:33:39 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script volume start vol_793f2fea1ecd5540c6c7b9011fef2fef
>Result: volume start: vol_793f2fea1ecd5540c6c7b9011fef2fef: success
>[heketi] INFO 2018/06/08 08:33:39 Create Volume succeeded
>[asynchttp] INFO 2018/06/08 08:33:39 asynchttp.go:292: Completed job a6b906276aeeaaac96c8168db7c988bb in 5.672020936s
>[negroni] Started GET /queue/a6b906276aeeaaac96c8168db7c988bb
>[negroni] Completed 303 See Other in 281.441Âµs
>[negroni] Started GET /volumes/793f2fea1ecd5540c6c7b9011fef2fef
>[negroni] Completed 200 OK in 4.281862ms
>[negroni] Started GET /clusters
>[negroni] Completed 200 OK in 208.003Âµs
>[negroni] Started GET /clusters/0a73c60efdd4673113b668afea101e6d
>[negroni] Completed 200 OK in 280.271Âµs
>[negroni] Started GET /volumes/793f2fea1ecd5540c6c7b9011fef2fef
>[negroni] Completed 200 OK in 987.406Âµs
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 628.178Âµs
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 1.607917ms
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 1.025898ms
>[negroni] Started GET /nodes/278bd6b4e16a8e62ef15aaae22e6abc1
>[negroni] Completed 200 OK in 1.30019ms
>[negroni] Started GET /nodes/70423cc0bcd044fe5ba8bbbd256a3e49
>[negroni] Completed 200 OK in 977.048Âµs
>[negroni] Started GET /nodes/d942fe6c0ee5691b7cc263968f97b650
>[negroni] Completed 200 OK in 819.679Âµs
>[negroni] Started POST /devices
>[heketi] INFO 2018/06/08 08:33:39 Adding device /dev/sdf to node 70423cc0bcd044fe5ba8bbbd256a3e49
>[negroni] Completed 202 Accepted in 10.38855ms
>[asynchttp] INFO 2018/06/08 08:33:39 asynchttp.go:288: Started job fb0fdb6d6bfe1bc1e72f1b72d34e734c
>[negroni] Started GET /queue/fb0fdb6d6bfe1bc1e72f1b72d34e734c
>[negroni] Completed 200 OK in 111.967Âµs
>[kubeexec] DEBUG 2018/06/08 08:33:40 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: pvcreate --metadatasize=128M --dataalignment=256K '/dev/sdf'
>Result:   Physical volume "/dev/sdf" successfully created.
>[kubeexec] DEBUG 2018/06/08 08:33:40 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: vgcreate --autobackup=n vg_b6293dbff56320d01ba8a795f86a2b5f /dev/sdf
>Result:   Volume group "vg_b6293dbff56320d01ba8a795f86a2b5f" successfully created
>[kubeexec] DEBUG 2018/06/08 08:33:40 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: vgdisplay -c vg_b6293dbff56320d01ba8a795f86a2b5f
>Result:   vg_b6293dbff56320d01ba8a795f86a2b5f:r/w:772:-1:0:0:0:-1:0:1:1:104722432:4096:25567:0:25567:OMEJM3-b3Xx-GZC1-dQSj-WHSM-XDDc-QCeiaG
>[cmdexec] DEBUG 2018/06/08 08:33:40 /src/github.com/heketi/heketi/executors/cmdexec/device.go:147: Size of /dev/sdf in dhcp46-122.lab.eng.blr.redhat.com is 104722432
>[heketi] INFO 2018/06/08 08:33:40 Added device /dev/sdf
>[asynchttp] INFO 2018/06/08 08:33:40 asynchttp.go:292: Completed job fb0fdb6d6bfe1bc1e72f1b72d34e734c in 653.134034ms
>[negroni] Started GET /queue/fb0fdb6d6bfe1bc1e72f1b72d34e734c
>[negroni] Completed 204 No Content in 182.131Âµs
>[negroni] Started GET /clusters
>[negroni] Completed 200 OK in 1.441963ms
>[negroni] Started GET /clusters/0a73c60efdd4673113b668afea101e6d
>[negroni] Completed 200 OK in 316.841Âµs
>[negroni] Started GET /volumes/793f2fea1ecd5540c6c7b9011fef2fef
>[negroni] Completed 200 OK in 1.492479ms
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 966.136Âµs
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 764.237Âµs
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 656.167Âµs
>[negroni] Started GET /nodes/278bd6b4e16a8e62ef15aaae22e6abc1
>[negroni] Completed 200 OK in 992.265Âµs
>[negroni] Started GET /nodes/70423cc0bcd044fe5ba8bbbd256a3e49
>[negroni] Completed 200 OK in 958.383Âµs
>[negroni] Started GET /nodes/d942fe6c0ee5691b7cc263968f97b650
>[negroni] Completed 200 OK in 726.622Âµs
>[negroni] Started POST /devices/b6293dbff56320d01ba8a795f86a2b5f/state
>[negroni] Completed 202 Accepted in 485.924Âµs
>[asynchttp] INFO 2018/06/08 08:33:41 asynchttp.go:288: Started job a857904fe3a2711216c5813f9db75cba
>[negroni] Started GET /queue/a857904fe3a2711216c5813f9db75cba
>[negroni] Completed 200 OK in 115.85Âµs
>[asynchttp] INFO 2018/06/08 08:33:41 asynchttp.go:292: Completed job a857904fe3a2711216c5813f9db75cba in 7.634885ms
>[negroni] Started GET /queue/a857904fe3a2711216c5813f9db75cba
>[negroni] Completed 204 No Content in 154.032Âµs
>[negroni] Started POST /devices/b6293dbff56320d01ba8a795f86a2b5f/state
>[negroni] Completed 202 Accepted in 1.693803ms
>[asynchttp] INFO 2018/06/08 08:33:42 asynchttp.go:288: Started job 310142380ed00bede5e869adf23a8b64
>[heketi] INFO 2018/06/08 08:33:42 Running Remove Device
>[negroni] Started GET /queue/310142380ed00bede5e869adf23a8b64
>[negroni] Completed 200 OK in 243.504Âµs
>[asynchttp] INFO 2018/06/08 08:33:42 asynchttp.go:292: Completed job 310142380ed00bede5e869adf23a8b64 in 9.18147ms
>[negroni] Started GET /queue/310142380ed00bede5e869adf23a8b64
>[negroni] Completed 204 No Content in 158.279Âµs
>[negroni] Started DELETE /devices/b6293dbff56320d01ba8a795f86a2b5f
>[heketi] INFO 2018/06/08 08:33:43 Deleting device b6293dbff56320d01ba8a795f86a2b5f on node 70423cc0bcd044fe5ba8bbbd256a3e49
>[negroni] Completed 202 Accepted in 1.400707ms
>[asynchttp] INFO 2018/06/08 08:33:43 asynchttp.go:288: Started job d1bc4ee485675b71a499e0e34583b5d4
>[negroni] Started GET /queue/d1bc4ee485675b71a499e0e34583b5d4
>[negroni] Completed 200 OK in 104.215Âµs
>[kubeexec] DEBUG 2018/06/08 08:33:43 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: vgremove vg_b6293dbff56320d01ba8a795f86a2b5f
>Result:   Volume group "vg_b6293dbff56320d01ba8a795f86a2b5f" successfully removed
>[kubeexec] DEBUG 2018/06/08 08:33:43 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: pvremove '/dev/sdf'
>Result:   Labels on physical volume "/dev/sdf" successfully wiped.
>[kubeexec] ERROR 2018/06/08 08:33:43 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [ls /var/lib/heketi/mounts/vg_b6293dbff56320d01ba8a795f86a2b5f] on glusterfs-storage-pg4xc: Err[command terminated with exit code 2]: Stdout []: Stderr [ls: cannot access /var/lib/heketi/mounts/vg_b6293dbff56320d01ba8a795f86a2b5f: No such file or directory
>]
>[heketi] INFO 2018/06/08 08:33:43 Deleted node [b6293dbff56320d01ba8a795f86a2b5f]
>[asynchttp] INFO 2018/06/08 08:33:43 asynchttp.go:292: Completed job d1bc4ee485675b71a499e0e34583b5d4 in 518.866636ms
>[negroni] Started GET /queue/d1bc4ee485675b71a499e0e34583b5d4
>[negroni] Completed 204 No Content in 150.003Âµs
>[negroni] Started GET /clusters
>[negroni] Completed 200 OK in 1.93406ms
>[negroni] Started GET /clusters/0a73c60efdd4673113b668afea101e6d
>[negroni] Completed 200 OK in 505.786Âµs
>[negroni] Started GET /volumes/793f2fea1ecd5540c6c7b9011fef2fef
>[negroni] Completed 200 OK in 1.884836ms
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 1.495029ms
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 1.191414ms
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 675.798Âµs
>[negroni] Started GET /nodes/278bd6b4e16a8e62ef15aaae22e6abc1
>[negroni] Completed 200 OK in 1.880039ms
>[negroni] Started GET /nodes/70423cc0bcd044fe5ba8bbbd256a3e49
>[negroni] Completed 200 OK in 1.535855ms
>[negroni] Started GET /nodes/d942fe6c0ee5691b7cc263968f97b650
>[negroni] Completed 200 OK in 1.264264ms
>[negroni] Started POST /volumes
>[heketi] INFO 2018/06/08 08:33:44 Allocating brick set #0
>[negroni] Completed 202 Accepted in 18.969419ms
>[asynchttp] INFO 2018/06/08 08:33:44 asynchttp.go:288: Started job b94ff76af50977cdc0c77fea87a14955
>[heketi] INFO 2018/06/08 08:33:44 Started async operation: Create Volume
>[heketi] INFO 2018/06/08 08:33:44 Creating brick 0f1d69bf8eee0f41a939f35c5a4e36e8
>[heketi] INFO 2018/06/08 08:33:44 Creating brick d8e106a809d43a339c3db048d0b81831
>[heketi] INFO 2018/06/08 08:33:44 Creating brick 2fdd0b36617fe22a539bf2bb950d7a8c
>[negroni] Started GET /queue/b94ff76af50977cdc0c77fea87a14955
>[negroni] Completed 200 OK in 138.224Âµs
>[kubeexec] DEBUG 2018/06/08 08:33:44 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir -p /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_0f1d69bf8eee0f41a939f35c5a4e36e8
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:33:44 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir -p /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_d8e106a809d43a339c3db048d0b81831
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:33:44 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir -p /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_2fdd0b36617fe22a539bf2bb950d7a8c
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:33:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvcreate --autobackup=n --poolmetadatasize 8192K --chunksize 256K --size 1048576K --thin vg_66a9af9f6bac95f9d8d556a2f14c29d3/tp_0f1d69bf8eee0f41a939f35c5a4e36e8 --virtualsize 1048576K --name brick_0f1d69bf8eee0f41a939f35c5a4e36e8
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_0f1d69bf8eee0f41a939f35c5a4e36e8" created.
>[kubeexec] DEBUG 2018/06/08 08:33:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvcreate --autobackup=n --poolmetadatasize 8192K --chunksize 256K --size 1048576K --thin vg_b861af9cc67f5bf8d5a8f9e38f56b14e/tp_d8e106a809d43a339c3db048d0b81831 --virtualsize 1048576K --name brick_d8e106a809d43a339c3db048d0b81831
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_d8e106a809d43a339c3db048d0b81831" created.
>[kubeexec] DEBUG 2018/06/08 08:33:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvcreate --autobackup=n --poolmetadatasize 8192K --chunksize 256K --size 1048576K --thin vg_96f1667f2f1ced2c5ef94772922be93b/tp_2fdd0b36617fe22a539bf2bb950d7a8c --virtualsize 1048576K --name brick_2fdd0b36617fe22a539bf2bb950d7a8c
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_2fdd0b36617fe22a539bf2bb950d7a8c" created.
>[kubeexec] DEBUG 2018/06/08 08:33:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_0f1d69bf8eee0f41a939f35c5a4e36e8
>Result: meta-data=/dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_0f1d69bf8eee0f41a939f35c5a4e36e8 isize=512    agcount=8, agsize=32768 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=262144, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:33:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_d8e106a809d43a339c3db048d0b81831
>Result: meta-data=/dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_d8e106a809d43a339c3db048d0b81831 isize=512    agcount=8, agsize=32768 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=262144, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:33:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: awk "BEGIN {print \"/dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_0f1d69bf8eee0f41a939f35c5a4e36e8 /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_0f1d69bf8eee0f41a939f35c5a4e36e8 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:33:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: awk "BEGIN {print \"/dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_d8e106a809d43a339c3db048d0b81831 /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_d8e106a809d43a339c3db048d0b81831 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:33:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_2fdd0b36617fe22a539bf2bb950d7a8c
>Result: meta-data=/dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_2fdd0b36617fe22a539bf2bb950d7a8c isize=512    agcount=8, agsize=32768 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=262144, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:33:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: awk "BEGIN {print \"/dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_2fdd0b36617fe22a539bf2bb950d7a8c /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_2fdd0b36617fe22a539bf2bb950d7a8c xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:33:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_d8e106a809d43a339c3db048d0b81831 /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_d8e106a809d43a339c3db048d0b81831
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:33:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_0f1d69bf8eee0f41a939f35c5a4e36e8 /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_0f1d69bf8eee0f41a939f35c5a4e36e8
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:33:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_d8e106a809d43a339c3db048d0b81831/brick
>Result: 
>[negroni] Started GET /queue/b94ff76af50977cdc0c77fea87a14955
>[negroni] Completed 200 OK in 134.575Âµs
>[kubeexec] DEBUG 2018/06/08 08:33:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_2fdd0b36617fe22a539bf2bb950d7a8c /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_2fdd0b36617fe22a539bf2bb950d7a8c
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:33:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_0f1d69bf8eee0f41a939f35c5a4e36e8/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:33:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_2fdd0b36617fe22a539bf2bb950d7a8c/brick
>Result: 
>[cmdexec] INFO 2018/06/08 08:33:45 Creating volume vol_e2b01dc5bde10ca95a8fa2b92a019ca3 replica 3
>[kubeexec] DEBUG 2018/06/08 08:33:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script volume create vol_e2b01dc5bde10ca95a8fa2b92a019ca3 replica 3 10.70.46.122:/var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_0f1d69bf8eee0f41a939f35c5a4e36e8/brick 10.70.47.76:/var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_2fdd0b36617fe22a539bf2bb950d7a8c/brick 10.70.46.187:/var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_d8e106a809d43a339c3db048d0b81831/brick
>Result: volume create: vol_e2b01dc5bde10ca95a8fa2b92a019ca3: success: please start the volume to access data
>[negroni] Started GET /queue/b94ff76af50977cdc0c77fea87a14955
>[negroni] Completed 200 OK in 155.287Âµs
>[negroni] Started GET /queue/b94ff76af50977cdc0c77fea87a14955
>[negroni] Completed 200 OK in 271.082Âµs
>[negroni] Started GET /queue/b94ff76af50977cdc0c77fea87a14955
>[negroni] Completed 200 OK in 143.62Âµs
>[kubeexec] DEBUG 2018/06/08 08:33:49 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script volume start vol_e2b01dc5bde10ca95a8fa2b92a019ca3
>Result: volume start: vol_e2b01dc5bde10ca95a8fa2b92a019ca3: success
>[heketi] INFO 2018/06/08 08:33:49 Create Volume succeeded
>[asynchttp] INFO 2018/06/08 08:33:49 asynchttp.go:292: Completed job b94ff76af50977cdc0c77fea87a14955 in 4.577582489s
>[negroni] Started GET /queue/b94ff76af50977cdc0c77fea87a14955
>[negroni] Completed 303 See Other in 186.461Âµs
>[negroni] Started GET /volumes/e2b01dc5bde10ca95a8fa2b92a019ca3
>[negroni] Completed 200 OK in 3.151584ms
>[negroni] Started DELETE /volumes/e2b01dc5bde10ca95a8fa2b92a019ca3
>[negroni] Completed 202 Accepted in 10.219959ms
>[asynchttp] INFO 2018/06/08 08:33:49 asynchttp.go:288: Started job 644371ffaefa640ed2e8f976a95e9dbb
>[heketi] INFO 2018/06/08 08:33:49 Started async operation: Delete Volume
>[negroni] Started GET /queue/644371ffaefa640ed2e8f976a95e9dbb
>[negroni] Completed 200 OK in 158.844Âµs
>[kubeexec] DEBUG 2018/06/08 08:33:49 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script snapshot list vol_e2b01dc5bde10ca95a8fa2b92a019ca3 --xml
>Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
><cliOutput>
>  <opRet>0</opRet>
>  <opErrno>0</opErrno>
>  <opErrstr/>
>  <snapList>
>    <count>0</count>
>  </snapList>
></cliOutput>
>[negroni] Started GET /queue/644371ffaefa640ed2e8f976a95e9dbb
>[negroni] Completed 200 OK in 257.151Âµs
>[negroni] Started GET /queue/644371ffaefa640ed2e8f976a95e9dbb
>[negroni] Completed 200 OK in 246.361Âµs
>[kubeexec] DEBUG 2018/06/08 08:33:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script volume stop vol_e2b01dc5bde10ca95a8fa2b92a019ca3 force
>Result: volume stop: vol_e2b01dc5bde10ca95a8fa2b92a019ca3: success
>[kubeexec] DEBUG 2018/06/08 08:33:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script volume delete vol_e2b01dc5bde10ca95a8fa2b92a019ca3
>Result: volume delete: vol_e2b01dc5bde10ca95a8fa2b92a019ca3: success
>[heketi] INFO 2018/06/08 08:33:52 Deleting brick 2fdd0b36617fe22a539bf2bb950d7a8c
>[heketi] INFO 2018/06/08 08:33:52 Deleting brick 0f1d69bf8eee0f41a939f35c5a4e36e8
>[heketi] INFO 2018/06/08 08:33:52 Deleting brick d8e106a809d43a339c3db048d0b81831
>[kubeexec] DEBUG 2018/06/08 08:33:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mount | grep -w /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_0f1d69bf8eee0f41a939f35c5a4e36e8 | cut -d" " -f1
>Result: /dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_0f1d69bf8eee0f41a939f35c5a4e36e8
>[kubeexec] DEBUG 2018/06/08 08:33:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mount | grep -w /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_d8e106a809d43a339c3db048d0b81831 | cut -d" " -f1
>Result: /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_d8e106a809d43a339c3db048d0b81831
>[kubeexec] DEBUG 2018/06/08 08:33:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mount | grep -w /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_2fdd0b36617fe22a539bf2bb950d7a8c | cut -d" " -f1
>Result: /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_2fdd0b36617fe22a539bf2bb950d7a8c
>[kubeexec] DEBUG 2018/06/08 08:33:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_0f1d69bf8eee0f41a939f35c5a4e36e8
>
>Result:   vg_66a9af9f6bac95f9d8d556a2f14c29d3/tp_0f1d69bf8eee0f41a939f35c5a4e36e8
>[kubeexec] DEBUG 2018/06/08 08:33:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_d8e106a809d43a339c3db048d0b81831
>
>Result:   vg_b861af9cc67f5bf8d5a8f9e38f56b14e/tp_d8e106a809d43a339c3db048d0b81831
>[negroni] Started GET /queue/644371ffaefa640ed2e8f976a95e9dbb
>[negroni] Completed 200 OK in 185.554Âµs
>[kubeexec] DEBUG 2018/06/08 08:33:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_2fdd0b36617fe22a539bf2bb950d7a8c
>
>Result:   vg_96f1667f2f1ced2c5ef94772922be93b/tp_2fdd0b36617fe22a539bf2bb950d7a8c
>[kubeexec] DEBUG 2018/06/08 08:33:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: umount /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_0f1d69bf8eee0f41a939f35c5a4e36e8
>Result: 
>[negroni] Started GET /queue/644371ffaefa640ed2e8f976a95e9dbb
>[negroni] Completed 200 OK in 135.775Âµs
>[kubeexec] DEBUG 2018/06/08 08:33:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: umount /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_d8e106a809d43a339c3db048d0b81831
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:33:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: umount /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_2fdd0b36617fe22a539bf2bb950d7a8c
>Result: 
>[negroni] Started GET /queue/644371ffaefa640ed2e8f976a95e9dbb
>[negroni] Completed 200 OK in 253.861Âµs
>[kubeexec] DEBUG 2018/06/08 08:33:55 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: sed -i.save "/brick_0f1d69bf8eee0f41a939f35c5a4e36e8/d" /var/lib/heketi/fstab
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:33:55 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: sed -i.save "/brick_d8e106a809d43a339c3db048d0b81831/d" /var/lib/heketi/fstab
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:33:55 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: sed -i.save "/brick_2fdd0b36617fe22a539bf2bb950d7a8c/d" /var/lib/heketi/fstab
>Result: 
>[negroni] Started GET /queue/644371ffaefa640ed2e8f976a95e9dbb
>[negroni] Completed 200 OK in 218.103Âµs
>[kubeexec] DEBUG 2018/06/08 08:33:56 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvremove --autobackup=n -f /dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_0f1d69bf8eee0f41a939f35c5a4e36e8
>
>Result:   Logical volume "brick_0f1d69bf8eee0f41a939f35c5a4e36e8" successfully removed
>[kubeexec] DEBUG 2018/06/08 08:33:56 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvremove --autobackup=n -f /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_d8e106a809d43a339c3db048d0b81831
>
>Result:   Logical volume "brick_d8e106a809d43a339c3db048d0b81831" successfully removed
>[kubeexec] DEBUG 2018/06/08 08:33:56 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvremove --autobackup=n -f /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_2fdd0b36617fe22a539bf2bb950d7a8c
>
>Result:   Logical volume "brick_2fdd0b36617fe22a539bf2bb950d7a8c" successfully removed
>[negroni] Started GET /queue/644371ffaefa640ed2e8f976a95e9dbb
>[negroni] Completed 200 OK in 271.221Âµs
>[kubeexec] DEBUG 2018/06/08 08:33:57 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvs --noheadings --options=thin_count   vg_66a9af9f6bac95f9d8d556a2f14c29d3/tp_0f1d69bf8eee0f41a939f35c5a4e36e8
>
>Result:        0
>[kubeexec] DEBUG 2018/06/08 08:33:57 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvs --noheadings --options=thin_count   vg_b861af9cc67f5bf8d5a8f9e38f56b14e/tp_d8e106a809d43a339c3db048d0b81831
>
>Result:        0
>[negroni] Started GET /queue/644371ffaefa640ed2e8f976a95e9dbb
>[negroni] Completed 200 OK in 161.997Âµs
>[kubeexec] DEBUG 2018/06/08 08:33:57 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvs --noheadings --options=thin_count   vg_96f1667f2f1ced2c5ef94772922be93b/tp_2fdd0b36617fe22a539bf2bb950d7a8c
>
>Result:        0
>[kubeexec] DEBUG 2018/06/08 08:33:58 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvremove --autobackup=n -f   vg_66a9af9f6bac95f9d8d556a2f14c29d3/tp_0f1d69bf8eee0f41a939f35c5a4e36e8
>
>Result:   Logical volume "tp_0f1d69bf8eee0f41a939f35c5a4e36e8" successfully removed
>[negroni] Started GET /queue/644371ffaefa640ed2e8f976a95e9dbb
>[negroni] Completed 200 OK in 145.637Âµs
>[kubeexec] DEBUG 2018/06/08 08:33:58 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvremove --autobackup=n -f   vg_b861af9cc67f5bf8d5a8f9e38f56b14e/tp_d8e106a809d43a339c3db048d0b81831
>
>Result:   Logical volume "tp_d8e106a809d43a339c3db048d0b81831" successfully removed
>[kubeexec] DEBUG 2018/06/08 08:33:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvremove --autobackup=n -f   vg_96f1667f2f1ced2c5ef94772922be93b/tp_2fdd0b36617fe22a539bf2bb950d7a8c
>
>Result:   Logical volume "tp_2fdd0b36617fe22a539bf2bb950d7a8c" successfully removed
>[kubeexec] DEBUG 2018/06/08 08:33:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: rmdir /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_0f1d69bf8eee0f41a939f35c5a4e36e8
>Result: 
>[negroni] Started GET /queue/644371ffaefa640ed2e8f976a95e9dbb
>[negroni] Completed 200 OK in 273.848Âµs
>[kubeexec] DEBUG 2018/06/08 08:34:00 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: rmdir /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_d8e106a809d43a339c3db048d0b81831
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:34:00 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: rmdir /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_2fdd0b36617fe22a539bf2bb950d7a8c
>Result: 
>[heketi] INFO 2018/06/08 08:34:00 Delete Volume succeeded
>[asynchttp] INFO 2018/06/08 08:34:00 asynchttp.go:292: Completed job 644371ffaefa640ed2e8f976a95e9dbb in 10.458558471s
>[negroni] Started GET /queue/644371ffaefa640ed2e8f976a95e9dbb
>[negroni] Completed 204 No Content in 177.206Âµs
>[negroni] Started POST /devices/b6293dbff56320d01ba8a795f86a2b5f/state
>[negroni] Completed 404 Not Found in 1.564239ms
>[negroni] Started DELETE /volumes/793f2fea1ecd5540c6c7b9011fef2fef
>[negroni] Completed 202 Accepted in 9.776324ms
>[asynchttp] INFO 2018/06/08 08:34:01 asynchttp.go:288: Started job fb69590d1d0f76ff40b6a5068a0651c3
>[heketi] INFO 2018/06/08 08:34:01 Started async operation: Delete Volume
>[negroni] Started GET /queue/fb69590d1d0f76ff40b6a5068a0651c3
>[negroni] Completed 200 OK in 124.772Âµs
>[kubeexec] DEBUG 2018/06/08 08:34:01 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: gluster --mode=script snapshot list vol_793f2fea1ecd5540c6c7b9011fef2fef --xml
>Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
><cliOutput>
>  <opRet>0</opRet>
>  <opErrno>0</opErrno>
>  <opErrstr/>
>  <snapList>
>    <count>0</count>
>  </snapList>
></cliOutput>
>[negroni] Started GET /queue/fb69590d1d0f76ff40b6a5068a0651c3
>[negroni] Completed 200 OK in 234.287Âµs
>[negroni] Started GET /queue/fb69590d1d0f76ff40b6a5068a0651c3
>[negroni] Completed 200 OK in 286.619Âµs
>[kubeexec] DEBUG 2018/06/08 08:34:03 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: gluster --mode=script volume stop vol_793f2fea1ecd5540c6c7b9011fef2fef force
>Result: volume stop: vol_793f2fea1ecd5540c6c7b9011fef2fef: success
>[kubeexec] DEBUG 2018/06/08 08:34:03 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: gluster --mode=script volume delete vol_793f2fea1ecd5540c6c7b9011fef2fef
>Result: volume delete: vol_793f2fea1ecd5540c6c7b9011fef2fef: success
>[heketi] INFO 2018/06/08 08:34:03 Deleting brick 989fc70ef70fd18a678f7fc65ac3a56d
>[heketi] INFO 2018/06/08 08:34:03 Deleting brick ce0cbbac0ac897ad4fe0a2d0068195ae
>[heketi] INFO 2018/06/08 08:34:03 Deleting brick 53eb288b3c12a575d4cbef2bce0292ef
>[kubeexec] DEBUG 2018/06/08 08:34:03 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mount | grep -w /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_53eb288b3c12a575d4cbef2bce0292ef | cut -d" " -f1
>Result: /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_53eb288b3c12a575d4cbef2bce0292ef
>[kubeexec] DEBUG 2018/06/08 08:34:03 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mount | grep -w /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_989fc70ef70fd18a678f7fc65ac3a56d | cut -d" " -f1
>Result: /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_989fc70ef70fd18a678f7fc65ac3a56d
>[kubeexec] DEBUG 2018/06/08 08:34:03 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mount | grep -w /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_ce0cbbac0ac897ad4fe0a2d0068195ae | cut -d" " -f1
>Result: /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_ce0cbbac0ac897ad4fe0a2d0068195ae
>[kubeexec] DEBUG 2018/06/08 08:34:04 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_53eb288b3c12a575d4cbef2bce0292ef
>
>Result:   vg_d389f0278a774bd7443a09af960961d8/tp_53eb288b3c12a575d4cbef2bce0292ef
>[negroni] Started GET /queue/fb69590d1d0f76ff40b6a5068a0651c3
>[negroni] Completed 200 OK in 238.084Âµs
>[kubeexec] DEBUG 2018/06/08 08:34:04 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_989fc70ef70fd18a678f7fc65ac3a56d
>
>Result:   vg_9394bc70699b006c5460c9f654cf345f/tp_989fc70ef70fd18a678f7fc65ac3a56d
>[kubeexec] DEBUG 2018/06/08 08:34:04 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_ce0cbbac0ac897ad4fe0a2d0068195ae
>
>Result:   vg_3a4297677881963e3f80124971d50eea/tp_ce0cbbac0ac897ad4fe0a2d0068195ae
>[kubeexec] DEBUG 2018/06/08 08:34:04 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: umount /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_53eb288b3c12a575d4cbef2bce0292ef
>Result: 
>[negroni] Started GET /queue/fb69590d1d0f76ff40b6a5068a0651c3
>[negroni] Completed 200 OK in 194.876Âµs
>[kubeexec] DEBUG 2018/06/08 08:34:05 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: umount /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_989fc70ef70fd18a678f7fc65ac3a56d
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:34:06 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: umount /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_ce0cbbac0ac897ad4fe0a2d0068195ae
>Result: 
>[negroni] Started GET /queue/fb69590d1d0f76ff40b6a5068a0651c3
>[negroni] Completed 200 OK in 285.727Âµs
>[kubeexec] DEBUG 2018/06/08 08:34:06 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: sed -i.save "/brick_53eb288b3c12a575d4cbef2bce0292ef/d" /var/lib/heketi/fstab
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:34:06 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: sed -i.save "/brick_989fc70ef70fd18a678f7fc65ac3a56d/d" /var/lib/heketi/fstab
>Result: 
>[negroni] Started GET /queue/fb69590d1d0f76ff40b6a5068a0651c3
>[negroni] Completed 200 OK in 267.254Âµs
>[kubeexec] DEBUG 2018/06/08 08:34:07 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: sed -i.save "/brick_ce0cbbac0ac897ad4fe0a2d0068195ae/d" /var/lib/heketi/fstab
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:34:07 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvremove --autobackup=n -f /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_53eb288b3c12a575d4cbef2bce0292ef
>
>Result:   Logical volume "brick_53eb288b3c12a575d4cbef2bce0292ef" successfully removed
>[kubeexec] DEBUG 2018/06/08 08:34:07 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvremove --autobackup=n -f /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_989fc70ef70fd18a678f7fc65ac3a56d
>
>Result:   Logical volume "brick_989fc70ef70fd18a678f7fc65ac3a56d" successfully removed
>[negroni] Started GET /queue/fb69590d1d0f76ff40b6a5068a0651c3
>[negroni] Completed 200 OK in 143.391Âµs
>[kubeexec] DEBUG 2018/06/08 08:34:08 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvremove --autobackup=n -f /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_ce0cbbac0ac897ad4fe0a2d0068195ae
>
>Result:   Logical volume "brick_ce0cbbac0ac897ad4fe0a2d0068195ae" successfully removed
>[kubeexec] DEBUG 2018/06/08 08:34:08 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvs --noheadings --options=thin_count   vg_d389f0278a774bd7443a09af960961d8/tp_53eb288b3c12a575d4cbef2bce0292ef
>
>Result:        0
>[kubeexec] DEBUG 2018/06/08 08:34:08 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvs --noheadings --options=thin_count   vg_9394bc70699b006c5460c9f654cf345f/tp_989fc70ef70fd18a678f7fc65ac3a56d
>
>Result:        0
>[negroni] Started GET /queue/fb69590d1d0f76ff40b6a5068a0651c3
>[negroni] Completed 200 OK in 168.769Âµs
>[kubeexec] DEBUG 2018/06/08 08:34:09 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvs --noheadings --options=thin_count   vg_3a4297677881963e3f80124971d50eea/tp_ce0cbbac0ac897ad4fe0a2d0068195ae
>
>Result:        0
>[kubeexec] DEBUG 2018/06/08 08:34:09 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvremove --autobackup=n -f   vg_d389f0278a774bd7443a09af960961d8/tp_53eb288b3c12a575d4cbef2bce0292ef
>
>Result:   Logical volume "tp_53eb288b3c12a575d4cbef2bce0292ef" successfully removed
>[kubeexec] DEBUG 2018/06/08 08:34:10 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvremove --autobackup=n -f   vg_9394bc70699b006c5460c9f654cf345f/tp_989fc70ef70fd18a678f7fc65ac3a56d
>
>Result:   Logical volume "tp_989fc70ef70fd18a678f7fc65ac3a56d" successfully removed
>[negroni] Started GET /queue/fb69590d1d0f76ff40b6a5068a0651c3
>[negroni] Completed 200 OK in 141.103Âµs
>[kubeexec] DEBUG 2018/06/08 08:34:10 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvremove --autobackup=n -f   vg_3a4297677881963e3f80124971d50eea/tp_ce0cbbac0ac897ad4fe0a2d0068195ae
>
>Result:   Logical volume "tp_ce0cbbac0ac897ad4fe0a2d0068195ae" successfully removed
>[kubeexec] DEBUG 2018/06/08 08:34:10 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: rmdir /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_53eb288b3c12a575d4cbef2bce0292ef
>Result: 
>[negroni] Started GET /queue/fb69590d1d0f76ff40b6a5068a0651c3
>[negroni] Completed 200 OK in 158.441Âµs
>[kubeexec] DEBUG 2018/06/08 08:34:11 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: rmdir /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_989fc70ef70fd18a678f7fc65ac3a56d
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:34:11 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: rmdir /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_ce0cbbac0ac897ad4fe0a2d0068195ae
>Result: 
>[heketi] INFO 2018/06/08 08:34:11 Delete Volume succeeded
>[asynchttp] INFO 2018/06/08 08:34:11 asynchttp.go:292: Completed job fb69590d1d0f76ff40b6a5068a0651c3 in 10.554691286s
>[negroni] Started GET /queue/fb69590d1d0f76ff40b6a5068a0651c3
>[negroni] Completed 204 No Content in 172.12Âµs
>[negroni] Started POST /volumes
>[heketi] INFO 2018/06/08 08:34:12 Allocating brick set #0
>[negroni] Completed 202 Accepted in 17.374374ms
>[asynchttp] INFO 2018/06/08 08:34:12 asynchttp.go:288: Started job 2bbaf93134b525b49c799f542f62ac0c
>[heketi] INFO 2018/06/08 08:34:12 Started async operation: Create Volume
>[heketi] INFO 2018/06/08 08:34:12 Creating brick 506f35d8abd5d43ae307f4211452c5d9
>[heketi] INFO 2018/06/08 08:34:12 Creating brick 3a7d32deb339b6fa432b0fcbc36ae94c
>[heketi] INFO 2018/06/08 08:34:12 Creating brick d59b610bd2c0773ffd302f1611ae4a02
>[negroni] Started GET /queue/2bbaf93134b525b49c799f542f62ac0c
>[negroni] Completed 200 OK in 150.676Âµs
>[kubeexec] DEBUG 2018/06/08 08:34:12 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir -p /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_d59b610bd2c0773ffd302f1611ae4a02
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:34:12 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir -p /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_3a7d32deb339b6fa432b0fcbc36ae94c
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:34:12 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvcreate --autobackup=n --poolmetadatasize 8192K --chunksize 256K --size 1048576K --thin vg_d389f0278a774bd7443a09af960961d8/tp_d59b610bd2c0773ffd302f1611ae4a02 --virtualsize 1048576K --name brick_d59b610bd2c0773ffd302f1611ae4a02
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_d59b610bd2c0773ffd302f1611ae4a02" created.
>[kubeexec] DEBUG 2018/06/08 08:34:12 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir -p /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_506f35d8abd5d43ae307f4211452c5d9
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:34:12 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_d59b610bd2c0773ffd302f1611ae4a02
>Result: meta-data=/dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_d59b610bd2c0773ffd302f1611ae4a02 isize=512    agcount=8, agsize=32768 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=262144, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:34:12 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvcreate --autobackup=n --poolmetadatasize 8192K --chunksize 256K --size 1048576K --thin vg_66a9af9f6bac95f9d8d556a2f14c29d3/tp_3a7d32deb339b6fa432b0fcbc36ae94c --virtualsize 1048576K --name brick_3a7d32deb339b6fa432b0fcbc36ae94c
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_3a7d32deb339b6fa432b0fcbc36ae94c" created.
>[kubeexec] DEBUG 2018/06/08 08:34:12 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: awk "BEGIN {print \"/dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_d59b610bd2c0773ffd302f1611ae4a02 /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_d59b610bd2c0773ffd302f1611ae4a02 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:34:13 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_3a7d32deb339b6fa432b0fcbc36ae94c
>Result: meta-data=/dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_3a7d32deb339b6fa432b0fcbc36ae94c isize=512    agcount=8, agsize=32768 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=262144, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:34:13 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_d59b610bd2c0773ffd302f1611ae4a02 /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_d59b610bd2c0773ffd302f1611ae4a02
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:34:13 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvcreate --autobackup=n --poolmetadatasize 8192K --chunksize 256K --size 1048576K --thin vg_9394bc70699b006c5460c9f654cf345f/tp_506f35d8abd5d43ae307f4211452c5d9 --virtualsize 1048576K --name brick_506f35d8abd5d43ae307f4211452c5d9
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_506f35d8abd5d43ae307f4211452c5d9" created.
>[kubeexec] DEBUG 2018/06/08 08:34:13 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: awk "BEGIN {print \"/dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_3a7d32deb339b6fa432b0fcbc36ae94c /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_3a7d32deb339b6fa432b0fcbc36ae94c xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:34:13 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_d59b610bd2c0773ffd302f1611ae4a02/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:34:13 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_506f35d8abd5d43ae307f4211452c5d9
>Result: meta-data=/dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_506f35d8abd5d43ae307f4211452c5d9 isize=512    agcount=8, agsize=32768 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=262144, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[negroni] Started GET /queue/2bbaf93134b525b49c799f542f62ac0c
>[negroni] Completed 200 OK in 170.06Âµs
>[kubeexec] DEBUG 2018/06/08 08:34:13 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_3a7d32deb339b6fa432b0fcbc36ae94c /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_3a7d32deb339b6fa432b0fcbc36ae94c
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:34:13 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: awk "BEGIN {print \"/dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_506f35d8abd5d43ae307f4211452c5d9 /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_506f35d8abd5d43ae307f4211452c5d9 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:34:13 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_3a7d32deb339b6fa432b0fcbc36ae94c/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:34:13 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_506f35d8abd5d43ae307f4211452c5d9 /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_506f35d8abd5d43ae307f4211452c5d9
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:34:13 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_506f35d8abd5d43ae307f4211452c5d9/brick
>Result: 
>[cmdexec] INFO 2018/06/08 08:34:13 Creating volume vol_79d83a099d1495f94123b0ca3fa0bac2 replica 3
>[kubeexec] DEBUG 2018/06/08 08:34:13 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script volume create vol_79d83a099d1495f94123b0ca3fa0bac2 replica 3 10.70.47.76:/var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_506f35d8abd5d43ae307f4211452c5d9/brick 10.70.46.187:/var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_d59b610bd2c0773ffd302f1611ae4a02/brick 10.70.46.122:/var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_3a7d32deb339b6fa432b0fcbc36ae94c/brick
>Result: volume create: vol_79d83a099d1495f94123b0ca3fa0bac2: success: please start the volume to access data
>[negroni] Started GET /queue/2bbaf93134b525b49c799f542f62ac0c
>[negroni] Completed 200 OK in 130.126Âµs
>[negroni] Started GET /queue/2bbaf93134b525b49c799f542f62ac0c
>[negroni] Completed 200 OK in 128.348Âµs
>[negroni] Started GET /queue/2bbaf93134b525b49c799f542f62ac0c
>[negroni] Completed 200 OK in 165.648Âµs
>[negroni] Started GET /queue/2bbaf93134b525b49c799f542f62ac0c
>[negroni] Completed 200 OK in 229.158Âµs
>[kubeexec] DEBUG 2018/06/08 08:34:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script volume start vol_79d83a099d1495f94123b0ca3fa0bac2
>Result: volume start: vol_79d83a099d1495f94123b0ca3fa0bac2: success
>[heketi] INFO 2018/06/08 08:34:18 Create Volume succeeded
>[asynchttp] INFO 2018/06/08 08:34:18 asynchttp.go:292: Completed job 2bbaf93134b525b49c799f542f62ac0c in 5.803275233s
>[negroni] Started GET /queue/2bbaf93134b525b49c799f542f62ac0c
>[negroni] Completed 303 See Other in 235.328Âµs
>[negroni] Started GET /volumes/79d83a099d1495f94123b0ca3fa0bac2
>[negroni] Completed 200 OK in 3.572477ms
>[negroni] Started DELETE /volumes/79d83a099d1495f94123b0ca3fa0bac2
>[negroni] Completed 202 Accepted in 14.139887ms
>[asynchttp] INFO 2018/06/08 08:34:18 asynchttp.go:288: Started job 7c1b072ba88366cedd6a5c3427092a14
>[heketi] INFO 2018/06/08 08:34:18 Started async operation: Delete Volume
>[negroni] Started GET /queue/7c1b072ba88366cedd6a5c3427092a14
>[negroni] Completed 200 OK in 148.66Âµs
>[kubeexec] DEBUG 2018/06/08 08:34:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script snapshot list vol_79d83a099d1495f94123b0ca3fa0bac2 --xml
>Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
><cliOutput>
>  <opRet>0</opRet>
>  <opErrno>0</opErrno>
>  <opErrstr/>
>  <snapList>
>    <count>0</count>
>  </snapList>
></cliOutput>
>[negroni] Started GET /queue/7c1b072ba88366cedd6a5c3427092a14
>[negroni] Completed 200 OK in 213.843Âµs
>[negroni] Started GET /queue/7c1b072ba88366cedd6a5c3427092a14
>[negroni] Completed 200 OK in 240.44Âµs
>[negroni] Started GET /queue/7c1b072ba88366cedd6a5c3427092a14
>[negroni] Completed 200 OK in 215.6Âµs
>[kubeexec] DEBUG 2018/06/08 08:34:22 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script volume stop vol_79d83a099d1495f94123b0ca3fa0bac2 force
>Result: volume stop: vol_79d83a099d1495f94123b0ca3fa0bac2: success
>[kubeexec] DEBUG 2018/06/08 08:34:22 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script volume delete vol_79d83a099d1495f94123b0ca3fa0bac2
>Result: volume delete: vol_79d83a099d1495f94123b0ca3fa0bac2: success
>[heketi] INFO 2018/06/08 08:34:22 Deleting brick 3a7d32deb339b6fa432b0fcbc36ae94c
>[heketi] INFO 2018/06/08 08:34:22 Deleting brick 506f35d8abd5d43ae307f4211452c5d9
>[heketi] INFO 2018/06/08 08:34:22 Deleting brick d59b610bd2c0773ffd302f1611ae4a02
>[kubeexec] DEBUG 2018/06/08 08:34:22 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mount | grep -w /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_3a7d32deb339b6fa432b0fcbc36ae94c | cut -d" " -f1
>Result: /dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_3a7d32deb339b6fa432b0fcbc36ae94c
>[kubeexec] DEBUG 2018/06/08 08:34:22 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mount | grep -w /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_d59b610bd2c0773ffd302f1611ae4a02 | cut -d" " -f1
>Result: /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_d59b610bd2c0773ffd302f1611ae4a02
>[kubeexec] DEBUG 2018/06/08 08:34:22 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mount | grep -w /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_506f35d8abd5d43ae307f4211452c5d9 | cut -d" " -f1
>Result: /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_506f35d8abd5d43ae307f4211452c5d9
>[negroni] Started GET /queue/7c1b072ba88366cedd6a5c3427092a14
>[negroni] Completed 200 OK in 155.613Âµs
>[kubeexec] DEBUG 2018/06/08 08:34:22 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_3a7d32deb339b6fa432b0fcbc36ae94c
>
>Result:   vg_66a9af9f6bac95f9d8d556a2f14c29d3/tp_3a7d32deb339b6fa432b0fcbc36ae94c
>[kubeexec] DEBUG 2018/06/08 08:34:22 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_d59b610bd2c0773ffd302f1611ae4a02
>
>Result:   vg_d389f0278a774bd7443a09af960961d8/tp_d59b610bd2c0773ffd302f1611ae4a02
>[kubeexec] DEBUG 2018/06/08 08:34:22 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_506f35d8abd5d43ae307f4211452c5d9
>
>Result:   vg_9394bc70699b006c5460c9f654cf345f/tp_506f35d8abd5d43ae307f4211452c5d9
>[negroni] Started GET /queue/7c1b072ba88366cedd6a5c3427092a14
>[negroni] Completed 200 OK in 201.088Âµs
>[kubeexec] DEBUG 2018/06/08 08:34:23 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: umount /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_3a7d32deb339b6fa432b0fcbc36ae94c
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:34:23 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: umount /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_d59b610bd2c0773ffd302f1611ae4a02
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:34:24 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: umount /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_506f35d8abd5d43ae307f4211452c5d9
>Result: 
>[negroni] Started GET /queue/7c1b072ba88366cedd6a5c3427092a14
>[negroni] Completed 200 OK in 197.278Âµs
>[kubeexec] DEBUG 2018/06/08 08:34:24 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: sed -i.save "/brick_3a7d32deb339b6fa432b0fcbc36ae94c/d" /var/lib/heketi/fstab
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:34:25 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: sed -i.save "/brick_d59b610bd2c0773ffd302f1611ae4a02/d" /var/lib/heketi/fstab
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:34:25 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: sed -i.save "/brick_506f35d8abd5d43ae307f4211452c5d9/d" /var/lib/heketi/fstab
>Result: 
>[negroni] Started GET /queue/7c1b072ba88366cedd6a5c3427092a14
>[negroni] Completed 200 OK in 185.416Âµs
>[kubeexec] DEBUG 2018/06/08 08:34:26 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvremove --autobackup=n -f /dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_3a7d32deb339b6fa432b0fcbc36ae94c
>
>Result:   Logical volume "brick_3a7d32deb339b6fa432b0fcbc36ae94c" successfully removed
>[kubeexec] DEBUG 2018/06/08 08:34:26 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvremove --autobackup=n -f /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_d59b610bd2c0773ffd302f1611ae4a02
>
>Result:   Logical volume "brick_d59b610bd2c0773ffd302f1611ae4a02" successfully removed
>[kubeexec] DEBUG 2018/06/08 08:34:26 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvremove --autobackup=n -f /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_506f35d8abd5d43ae307f4211452c5d9
>
>Result:   Logical volume "brick_506f35d8abd5d43ae307f4211452c5d9" successfully removed
>[negroni] Started GET /queue/7c1b072ba88366cedd6a5c3427092a14
>[negroni] Completed 200 OK in 202.582Âµs
>[kubeexec] DEBUG 2018/06/08 08:34:27 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvs --noheadings --options=thin_count   vg_66a9af9f6bac95f9d8d556a2f14c29d3/tp_3a7d32deb339b6fa432b0fcbc36ae94c
>
>Result:        0
>[kubeexec] DEBUG 2018/06/08 08:34:27 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvs --noheadings --options=thin_count   vg_d389f0278a774bd7443a09af960961d8/tp_d59b610bd2c0773ffd302f1611ae4a02
>
>Result:        0
>[negroni] Started GET /queue/7c1b072ba88366cedd6a5c3427092a14
>[negroni] Completed 200 OK in 212.534Âµs
>[kubeexec] DEBUG 2018/06/08 08:34:27 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvs --noheadings --options=thin_count   vg_9394bc70699b006c5460c9f654cf345f/tp_506f35d8abd5d43ae307f4211452c5d9
>
>Result:        0
>[negroni] Started GET /queue/7c1b072ba88366cedd6a5c3427092a14
>[negroni] Completed 200 OK in 131.39Âµs
>[kubeexec] DEBUG 2018/06/08 08:34:28 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvremove --autobackup=n -f   vg_66a9af9f6bac95f9d8d556a2f14c29d3/tp_3a7d32deb339b6fa432b0fcbc36ae94c
>
>Result:   Logical volume "tp_3a7d32deb339b6fa432b0fcbc36ae94c" successfully removed
>[kubeexec] DEBUG 2018/06/08 08:34:28 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvremove --autobackup=n -f   vg_d389f0278a774bd7443a09af960961d8/tp_d59b610bd2c0773ffd302f1611ae4a02
>
>Result:   Logical volume "tp_d59b610bd2c0773ffd302f1611ae4a02" successfully removed
>[kubeexec] DEBUG 2018/06/08 08:34:28 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvremove --autobackup=n -f   vg_9394bc70699b006c5460c9f654cf345f/tp_506f35d8abd5d43ae307f4211452c5d9
>
>Result:   Logical volume "tp_506f35d8abd5d43ae307f4211452c5d9" successfully removed
>[negroni] Started GET /queue/7c1b072ba88366cedd6a5c3427092a14
>[negroni] Completed 200 OK in 203.682Âµs
>[kubeexec] DEBUG 2018/06/08 08:34:29 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: rmdir /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_3a7d32deb339b6fa432b0fcbc36ae94c
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:34:29 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: rmdir /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_d59b610bd2c0773ffd302f1611ae4a02
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:34:30 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: rmdir /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_506f35d8abd5d43ae307f4211452c5d9
>Result: 
>[heketi] INFO 2018/06/08 08:34:30 Delete Volume succeeded
>[asynchttp] INFO 2018/06/08 08:34:30 asynchttp.go:292: Completed job 7c1b072ba88366cedd6a5c3427092a14 in 11.572650265s
>[negroni] Started GET /queue/7c1b072ba88366cedd6a5c3427092a14
>[negroni] Completed 204 No Content in 224.231Âµs
>[negroni] Started POST /volumes
>[heketi] INFO 2018/06/08 08:34:30 Allocating brick set #0
>[heketi] INFO 2018/06/08 08:34:30 Allocating brick set #0
>[heketi] INFO 2018/06/08 08:34:30 Allocating brick set #1
>[heketi] INFO 2018/06/08 08:34:30 Allocating brick set #0
>[heketi] INFO 2018/06/08 08:34:30 Allocating brick set #1
>[heketi] INFO 2018/06/08 08:34:30 Allocating brick set #2
>[heketi] INFO 2018/06/08 08:34:30 Allocating brick set #3
>[heketi] INFO 2018/06/08 08:34:30 Allocating brick set #0
>[heketi] INFO 2018/06/08 08:34:30 Allocating brick set #1
>[heketi] INFO 2018/06/08 08:34:30 Allocating brick set #2
>[heketi] INFO 2018/06/08 08:34:30 Allocating brick set #3
>[heketi] INFO 2018/06/08 08:34:30 Allocating brick set #4
>[heketi] INFO 2018/06/08 08:34:30 Allocating brick set #5
>[heketi] INFO 2018/06/08 08:34:30 Allocating brick set #6
>[heketi] ERROR 2018/06/08 08:34:30 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:1167: Create Volume Build Failed: No space
>[negroni] Completed 500 Internal Server Error in 20.545628ms
>[negroni] Started POST /volumes
>[heketi] INFO 2018/06/08 08:34:30 Allocating brick set #0
>[negroni] Completed 202 Accepted in 15.097946ms
>[asynchttp] INFO 2018/06/08 08:34:30 asynchttp.go:288: Started job ec8ee4e8fce2ab6e4876ed8c4f16b20f
>[heketi] INFO 2018/06/08 08:34:30 Started async operation: Create Volume
>[negroni] Started GET /queue/ec8ee4e8fce2ab6e4876ed8c4f16b20f
>[heketi] INFO 2018/06/08 08:34:30 Creating brick 50f7fa4e28c0ada62aee23b461d4a57b
>[negroni] Completed 200 OK in 149.632Âµs
>[heketi] INFO 2018/06/08 08:34:30 Creating brick c91ff80db7a76c089815fbb5acd3b359
>[heketi] INFO 2018/06/08 08:34:30 Creating brick 3ffb812f7f348d836df1be2db94be607
>[kubeexec] DEBUG 2018/06/08 08:34:30 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir -p /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_3ffb812f7f348d836df1be2db94be607
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:34:31 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir -p /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_50f7fa4e28c0ada62aee23b461d4a57b
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:34:31 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir -p /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_c91ff80db7a76c089815fbb5acd3b359
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:34:31 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvcreate --autobackup=n --poolmetadatasize 8192K --chunksize 256K --size 1048576K --thin vg_b861af9cc67f5bf8d5a8f9e38f56b14e/tp_3ffb812f7f348d836df1be2db94be607 --virtualsize 1048576K --name brick_3ffb812f7f348d836df1be2db94be607
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_3ffb812f7f348d836df1be2db94be607" created.
>[kubeexec] DEBUG 2018/06/08 08:34:31 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvcreate --autobackup=n --poolmetadatasize 8192K --chunksize 256K --size 1048576K --thin vg_96f1667f2f1ced2c5ef94772922be93b/tp_50f7fa4e28c0ada62aee23b461d4a57b --virtualsize 1048576K --name brick_50f7fa4e28c0ada62aee23b461d4a57b
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_50f7fa4e28c0ada62aee23b461d4a57b" created.
>[kubeexec] DEBUG 2018/06/08 08:34:31 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_3ffb812f7f348d836df1be2db94be607
>Result: meta-data=/dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_3ffb812f7f348d836df1be2db94be607 isize=512    agcount=8, agsize=32768 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=262144, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:34:31 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_50f7fa4e28c0ada62aee23b461d4a57b
>Result: meta-data=/dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_50f7fa4e28c0ada62aee23b461d4a57b isize=512    agcount=8, agsize=32768 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=262144, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:34:31 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: awk "BEGIN {print \"/dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_3ffb812f7f348d836df1be2db94be607 /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_3ffb812f7f348d836df1be2db94be607 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:34:31 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvcreate --autobackup=n --poolmetadatasize 8192K --chunksize 256K --size 1048576K --thin vg_3a4297677881963e3f80124971d50eea/tp_c91ff80db7a76c089815fbb5acd3b359 --virtualsize 1048576K --name brick_c91ff80db7a76c089815fbb5acd3b359
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_c91ff80db7a76c089815fbb5acd3b359" created.
>[kubeexec] DEBUG 2018/06/08 08:34:31 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: awk "BEGIN {print \"/dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_50f7fa4e28c0ada62aee23b461d4a57b /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_50f7fa4e28c0ada62aee23b461d4a57b xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:34:31 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_3ffb812f7f348d836df1be2db94be607 /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_3ffb812f7f348d836df1be2db94be607
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:34:31 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_c91ff80db7a76c089815fbb5acd3b359
>Result: meta-data=/dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_c91ff80db7a76c089815fbb5acd3b359 isize=512    agcount=8, agsize=32768 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=262144, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:34:31 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_50f7fa4e28c0ada62aee23b461d4a57b /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_50f7fa4e28c0ada62aee23b461d4a57b
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:34:31 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_3ffb812f7f348d836df1be2db94be607/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:34:31 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: awk "BEGIN {print \"/dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_c91ff80db7a76c089815fbb5acd3b359 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_c91ff80db7a76c089815fbb5acd3b359 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[negroni] Started GET /queue/ec8ee4e8fce2ab6e4876ed8c4f16b20f
>[negroni] Completed 200 OK in 222.959Âµs
>[kubeexec] DEBUG 2018/06/08 08:34:31 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_50f7fa4e28c0ada62aee23b461d4a57b/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:34:31 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_c91ff80db7a76c089815fbb5acd3b359 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_c91ff80db7a76c089815fbb5acd3b359
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:34:32 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_c91ff80db7a76c089815fbb5acd3b359/brick
>Result: 
>[cmdexec] INFO 2018/06/08 08:34:32 Creating volume vol_93defdf227537c8facf542c2a30034c5 replica 3
>[kubeexec] DEBUG 2018/06/08 08:34:32 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script volume create vol_93defdf227537c8facf542c2a30034c5 replica 3 10.70.47.76:/var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_50f7fa4e28c0ada62aee23b461d4a57b/brick 10.70.46.187:/var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_3ffb812f7f348d836df1be2db94be607/brick 10.70.46.122:/var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_c91ff80db7a76c089815fbb5acd3b359/brick
>Result: volume create: vol_93defdf227537c8facf542c2a30034c5: success: please start the volume to access data
>[negroni] Started GET /queue/ec8ee4e8fce2ab6e4876ed8c4f16b20f
>[negroni] Completed 200 OK in 131.596Âµs
>[negroni] Started GET /queue/ec8ee4e8fce2ab6e4876ed8c4f16b20f
>[negroni] Completed 200 OK in 149.645Âµs
>[negroni] Started GET /queue/ec8ee4e8fce2ab6e4876ed8c4f16b20f
>[negroni] Completed 200 OK in 164.553Âµs
>[negroni] Started GET /queue/ec8ee4e8fce2ab6e4876ed8c4f16b20f
>[negroni] Completed 200 OK in 171.589Âµs
>[kubeexec] DEBUG 2018/06/08 08:34:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script volume start vol_93defdf227537c8facf542c2a30034c5
>Result: volume start: vol_93defdf227537c8facf542c2a30034c5: success
>[heketi] INFO 2018/06/08 08:34:36 Create Volume succeeded
>[asynchttp] INFO 2018/06/08 08:34:36 asynchttp.go:292: Completed job ec8ee4e8fce2ab6e4876ed8c4f16b20f in 5.709673516s
>[negroni] Started GET /queue/ec8ee4e8fce2ab6e4876ed8c4f16b20f
>[negroni] Completed 303 See Other in 297.284Âµs
>[negroni] Started GET /volumes/93defdf227537c8facf542c2a30034c5
>[negroni] Completed 200 OK in 3.673265ms
>[negroni] Started POST /volumes/93defdf227537c8facf542c2a30034c5/expand
>[heketi] INFO 2018/06/08 08:34:36 Allocating brick set #0
>[negroni] Completed 202 Accepted in 16.658397ms
>[asynchttp] INFO 2018/06/08 08:34:36 asynchttp.go:288: Started job 6652127cae25ca9516aead3a36fab9ac
>[heketi] INFO 2018/06/08 08:34:36 Started async operation: Expand Volume
>[heketi] INFO 2018/06/08 08:34:36 Creating brick 0fe2e9af9fd76f4fccdcf0e37a1f89ff
>[heketi] INFO 2018/06/08 08:34:36 Creating brick a0e0349eea524aa0f4d3ab0a34fa1304
>[negroni] Started GET /queue/6652127cae25ca9516aead3a36fab9ac
>[negroni] Completed 200 OK in 112.114Âµs
>[heketi] INFO 2018/06/08 08:34:36 Creating brick 5bec7c663a12e79390ec9be885d88a1d
>[kubeexec] DEBUG 2018/06/08 08:34:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir -p /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_a0e0349eea524aa0f4d3ab0a34fa1304
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:34:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir -p /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_5bec7c663a12e79390ec9be885d88a1d
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:34:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir -p /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_0fe2e9af9fd76f4fccdcf0e37a1f89ff
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:34:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_3a4297677881963e3f80124971d50eea/tp_0fe2e9af9fd76f4fccdcf0e37a1f89ff --virtualsize 2097152K --name brick_0fe2e9af9fd76f4fccdcf0e37a1f89ff
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_0fe2e9af9fd76f4fccdcf0e37a1f89ff" created.
>[kubeexec] DEBUG 2018/06/08 08:34:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_d389f0278a774bd7443a09af960961d8/tp_a0e0349eea524aa0f4d3ab0a34fa1304 --virtualsize 2097152K --name brick_a0e0349eea524aa0f4d3ab0a34fa1304
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_a0e0349eea524aa0f4d3ab0a34fa1304" created.
>[kubeexec] DEBUG 2018/06/08 08:34:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_9394bc70699b006c5460c9f654cf345f/tp_5bec7c663a12e79390ec9be885d88a1d --virtualsize 2097152K --name brick_5bec7c663a12e79390ec9be885d88a1d
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_5bec7c663a12e79390ec9be885d88a1d" created.
>[kubeexec] DEBUG 2018/06/08 08:34:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_0fe2e9af9fd76f4fccdcf0e37a1f89ff
>Result: meta-data=/dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_0fe2e9af9fd76f4fccdcf0e37a1f89ff isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:34:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_5bec7c663a12e79390ec9be885d88a1d
>Result: meta-data=/dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_5bec7c663a12e79390ec9be885d88a1d isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:34:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_a0e0349eea524aa0f4d3ab0a34fa1304
>Result: meta-data=/dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_a0e0349eea524aa0f4d3ab0a34fa1304 isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:34:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: awk "BEGIN {print \"/dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_0fe2e9af9fd76f4fccdcf0e37a1f89ff /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_0fe2e9af9fd76f4fccdcf0e37a1f89ff xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:34:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: awk "BEGIN {print \"/dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_a0e0349eea524aa0f4d3ab0a34fa1304 /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_a0e0349eea524aa0f4d3ab0a34fa1304 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:34:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: awk "BEGIN {print \"/dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_5bec7c663a12e79390ec9be885d88a1d /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_5bec7c663a12e79390ec9be885d88a1d xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:34:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_0fe2e9af9fd76f4fccdcf0e37a1f89ff /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_0fe2e9af9fd76f4fccdcf0e37a1f89ff
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:34:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_a0e0349eea524aa0f4d3ab0a34fa1304 /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_a0e0349eea524aa0f4d3ab0a34fa1304
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:34:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_5bec7c663a12e79390ec9be885d88a1d /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_5bec7c663a12e79390ec9be885d88a1d
>Result: 
>[negroni] Started GET /queue/6652127cae25ca9516aead3a36fab9ac
>[negroni] Completed 200 OK in 197.518Âµs
>[kubeexec] DEBUG 2018/06/08 08:34:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_0fe2e9af9fd76f4fccdcf0e37a1f89ff/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:34:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_a0e0349eea524aa0f4d3ab0a34fa1304/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:34:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_5bec7c663a12e79390ec9be885d88a1d/brick
>Result: 
>[negroni] Started GET /queue/6652127cae25ca9516aead3a36fab9ac
>[negroni] Completed 200 OK in 130.119Âµs
>[negroni] Started GET /queue/6652127cae25ca9516aead3a36fab9ac
>[negroni] Completed 200 OK in 133.441Âµs
>[negroni] Started GET /queue/6652127cae25ca9516aead3a36fab9ac
>[negroni] Completed 200 OK in 125.648Âµs
>[negroni] Started GET /queue/6652127cae25ca9516aead3a36fab9ac
>[negroni] Completed 200 OK in 191.362Âµs
>[kubeexec] DEBUG 2018/06/08 08:34:42 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script volume add-brick vol_93defdf227537c8facf542c2a30034c5 10.70.46.122:/var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_0fe2e9af9fd76f4fccdcf0e37a1f89ff/brick 10.70.47.76:/var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_5bec7c663a12e79390ec9be885d88a1d/brick 10.70.46.187:/var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_a0e0349eea524aa0f4d3ab0a34fa1304/brick
>Result: volume add-brick: success
>[negroni] Started GET /queue/6652127cae25ca9516aead3a36fab9ac
>[negroni] Completed 200 OK in 223.595Âµs
>[negroni] Started GET /queue/6652127cae25ca9516aead3a36fab9ac
>[negroni] Completed 200 OK in 133.99Âµs
>[negroni] Started GET /queue/6652127cae25ca9516aead3a36fab9ac
>[negroni] Completed 200 OK in 170.178Âµs
>[negroni] Started GET /queue/6652127cae25ca9516aead3a36fab9ac
>[negroni] Completed 200 OK in 331.391Âµs
>[negroni] Started GET /queue/6652127cae25ca9516aead3a36fab9ac
>[negroni] Completed 200 OK in 142.35Âµs
>[negroni] Started GET /queue/6652127cae25ca9516aead3a36fab9ac
>[negroni] Completed 200 OK in 173.185Âµs
>[negroni] Started GET /queue/6652127cae25ca9516aead3a36fab9ac
>[negroni] Completed 200 OK in 216.519Âµs
>[negroni] Started GET /queue/6652127cae25ca9516aead3a36fab9ac
>[negroni] Completed 200 OK in 150.373Âµs
>[negroni] Started GET /queue/6652127cae25ca9516aead3a36fab9ac
>[negroni] Completed 200 OK in 180.087Âµs
>[negroni] Started GET /queue/6652127cae25ca9516aead3a36fab9ac
>[negroni] Completed 200 OK in 148.493Âµs
>[kubeexec] DEBUG 2018/06/08 08:34:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script volume rebalance vol_93defdf227537c8facf542c2a30034c5 start
>Result: volume rebalance: vol_93defdf227537c8facf542c2a30034c5: success: Rebalance on vol_93defdf227537c8facf542c2a30034c5 has been started successfully. Use rebalance status command to check status of the rebalance process.
>ID: 1275029d-65a4-46a1-b8ea-a86ac0817ba6
>
>[heketi] INFO 2018/06/08 08:34:52 Expand Volume succeeded
>[asynchttp] INFO 2018/06/08 08:34:52 asynchttp.go:292: Completed job 6652127cae25ca9516aead3a36fab9ac in 15.731453773s
>[negroni] Started GET /queue/6652127cae25ca9516aead3a36fab9ac
>[negroni] Completed 303 See Other in 265.694Âµs
>[negroni] Started GET /volumes/93defdf227537c8facf542c2a30034c5
>[negroni] Completed 200 OK in 4.711108ms
>[negroni] Started GET /volumes/93defdf227537c8facf542c2a30034c5
>[negroni] Completed 200 OK in 1.630447ms
>[negroni] Started DELETE /volumes/93defdf227537c8facf542c2a30034c5
>[negroni] Completed 202 Accepted in 10.753205ms
>[asynchttp] INFO 2018/06/08 08:34:53 asynchttp.go:288: Started job cc8c30c83cc8de532e15e823f1889908
>[heketi] INFO 2018/06/08 08:34:53 Started async operation: Delete Volume
>[negroni] Started GET /queue/cc8c30c83cc8de532e15e823f1889908
>[negroni] Completed 200 OK in 111.942Âµs
>[kubeexec] DEBUG 2018/06/08 08:34:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script snapshot list vol_93defdf227537c8facf542c2a30034c5 --xml
>Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
><cliOutput>
>  <opRet>0</opRet>
>  <opErrno>0</opErrno>
>  <opErrstr/>
>  <snapList>
>    <count>0</count>
>  </snapList>
></cliOutput>
>[negroni] Started GET /queue/cc8c30c83cc8de532e15e823f1889908
>[negroni] Completed 200 OK in 162.665Âµs
>[negroni] Started GET /queue/cc8c30c83cc8de532e15e823f1889908
>[negroni] Completed 200 OK in 268.718Âµs
>[kubeexec] DEBUG 2018/06/08 08:34:55 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script volume stop vol_93defdf227537c8facf542c2a30034c5 force
>Result: volume stop: vol_93defdf227537c8facf542c2a30034c5: success
>[kubeexec] DEBUG 2018/06/08 08:34:56 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script volume delete vol_93defdf227537c8facf542c2a30034c5
>Result: volume delete: vol_93defdf227537c8facf542c2a30034c5: success
>[heketi] INFO 2018/06/08 08:34:56 Deleting brick a0e0349eea524aa0f4d3ab0a34fa1304
>[heketi] INFO 2018/06/08 08:34:56 Deleting brick 0fe2e9af9fd76f4fccdcf0e37a1f89ff
>[heketi] INFO 2018/06/08 08:34:56 Deleting brick 5bec7c663a12e79390ec9be885d88a1d
>[heketi] INFO 2018/06/08 08:34:56 Deleting brick c91ff80db7a76c089815fbb5acd3b359
>[heketi] INFO 2018/06/08 08:34:56 Deleting brick 3ffb812f7f348d836df1be2db94be607
>[heketi] INFO 2018/06/08 08:34:56 Deleting brick 50f7fa4e28c0ada62aee23b461d4a57b
>[kubeexec] DEBUG 2018/06/08 08:34:56 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mount | grep -w /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_0fe2e9af9fd76f4fccdcf0e37a1f89ff | cut -d" " -f1
>Result: /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_0fe2e9af9fd76f4fccdcf0e37a1f89ff
>[kubeexec] DEBUG 2018/06/08 08:34:56 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mount | grep -w /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_5bec7c663a12e79390ec9be885d88a1d | cut -d" " -f1
>Result: /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_5bec7c663a12e79390ec9be885d88a1d
>[negroni] Started GET /queue/cc8c30c83cc8de532e15e823f1889908
>[negroni] Completed 200 OK in 155.619Âµs
>[kubeexec] DEBUG 2018/06/08 08:34:56 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mount | grep -w /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_a0e0349eea524aa0f4d3ab0a34fa1304 | cut -d" " -f1
>Result: /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_a0e0349eea524aa0f4d3ab0a34fa1304
>[kubeexec] DEBUG 2018/06/08 08:34:56 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mount | grep -w /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_c91ff80db7a76c089815fbb5acd3b359 | cut -d" " -f1
>Result: /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_c91ff80db7a76c089815fbb5acd3b359
>[kubeexec] DEBUG 2018/06/08 08:34:56 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mount | grep -w /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_50f7fa4e28c0ada62aee23b461d4a57b | cut -d" " -f1
>Result: /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_50f7fa4e28c0ada62aee23b461d4a57b
>[kubeexec] DEBUG 2018/06/08 08:34:56 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mount | grep -w /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_3ffb812f7f348d836df1be2db94be607 | cut -d" " -f1
>Result: /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_3ffb812f7f348d836df1be2db94be607
>[negroni] Started GET /queue/cc8c30c83cc8de532e15e823f1889908
>[negroni] Completed 200 OK in 229.858Âµs
>[kubeexec] DEBUG 2018/06/08 08:34:57 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_0fe2e9af9fd76f4fccdcf0e37a1f89ff
>
>Result:   vg_3a4297677881963e3f80124971d50eea/tp_0fe2e9af9fd76f4fccdcf0e37a1f89ff
>[kubeexec] DEBUG 2018/06/08 08:34:57 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_5bec7c663a12e79390ec9be885d88a1d
>
>Result:   vg_9394bc70699b006c5460c9f654cf345f/tp_5bec7c663a12e79390ec9be885d88a1d
>[kubeexec] DEBUG 2018/06/08 08:34:57 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_a0e0349eea524aa0f4d3ab0a34fa1304
>
>Result:   vg_d389f0278a774bd7443a09af960961d8/tp_a0e0349eea524aa0f4d3ab0a34fa1304
>[negroni] Started GET /queue/cc8c30c83cc8de532e15e823f1889908
>[negroni] Completed 200 OK in 154.737Âµs
>[kubeexec] DEBUG 2018/06/08 08:34:58 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_c91ff80db7a76c089815fbb5acd3b359
>
>Result:   vg_3a4297677881963e3f80124971d50eea/tp_c91ff80db7a76c089815fbb5acd3b359
>[kubeexec] DEBUG 2018/06/08 08:34:58 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_50f7fa4e28c0ada62aee23b461d4a57b
>
>Result:   vg_96f1667f2f1ced2c5ef94772922be93b/tp_50f7fa4e28c0ada62aee23b461d4a57b
>[kubeexec] DEBUG 2018/06/08 08:34:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_3ffb812f7f348d836df1be2db94be607
>
>Result:   vg_b861af9cc67f5bf8d5a8f9e38f56b14e/tp_3ffb812f7f348d836df1be2db94be607
>[negroni] Started GET /queue/cc8c30c83cc8de532e15e823f1889908
>[negroni] Completed 200 OK in 250.558Âµs
>[kubeexec] DEBUG 2018/06/08 08:34:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: umount /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_0fe2e9af9fd76f4fccdcf0e37a1f89ff
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:35:00 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: umount /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_5bec7c663a12e79390ec9be885d88a1d
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:35:00 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: umount /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_a0e0349eea524aa0f4d3ab0a34fa1304
>Result: 
>[negroni] Started GET /queue/cc8c30c83cc8de532e15e823f1889908
>[negroni] Completed 200 OK in 203.331Âµs
>[kubeexec] DEBUG 2018/06/08 08:35:01 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: umount /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_c91ff80db7a76c089815fbb5acd3b359
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:35:01 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: umount /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_50f7fa4e28c0ada62aee23b461d4a57b
>Result: 
>[negroni] Started GET /queue/cc8c30c83cc8de532e15e823f1889908
>[negroni] Completed 200 OK in 145.776Âµs
>[kubeexec] DEBUG 2018/06/08 08:35:01 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: umount /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_3ffb812f7f348d836df1be2db94be607
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:35:02 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: sed -i.save "/brick_0fe2e9af9fd76f4fccdcf0e37a1f89ff/d" /var/lib/heketi/fstab
>Result: 
>[negroni] Started GET /queue/cc8c30c83cc8de532e15e823f1889908
>[negroni] Completed 200 OK in 176.777Âµs
>[kubeexec] DEBUG 2018/06/08 08:35:02 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: sed -i.save "/brick_5bec7c663a12e79390ec9be885d88a1d/d" /var/lib/heketi/fstab
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:35:02 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: sed -i.save "/brick_a0e0349eea524aa0f4d3ab0a34fa1304/d" /var/lib/heketi/fstab
>Result: 
>[negroni] Started GET /queue/cc8c30c83cc8de532e15e823f1889908
>[negroni] Completed 200 OK in 265.898Âµs
>[kubeexec] DEBUG 2018/06/08 08:35:03 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: sed -i.save "/brick_c91ff80db7a76c089815fbb5acd3b359/d" /var/lib/heketi/fstab
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:35:03 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: sed -i.save "/brick_50f7fa4e28c0ada62aee23b461d4a57b/d" /var/lib/heketi/fstab
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:35:03 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: sed -i.save "/brick_3ffb812f7f348d836df1be2db94be607/d" /var/lib/heketi/fstab
>Result: 
>[negroni] Started GET /queue/cc8c30c83cc8de532e15e823f1889908
>[negroni] Completed 200 OK in 220.631Âµs
>[kubeexec] DEBUG 2018/06/08 08:35:04 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvremove --autobackup=n -f /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_0fe2e9af9fd76f4fccdcf0e37a1f89ff
>
>Result:   Logical volume "brick_0fe2e9af9fd76f4fccdcf0e37a1f89ff" successfully removed
>[kubeexec] DEBUG 2018/06/08 08:35:04 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvremove --autobackup=n -f /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_5bec7c663a12e79390ec9be885d88a1d
>
>Result:   Logical volume "brick_5bec7c663a12e79390ec9be885d88a1d" successfully removed
>[kubeexec] DEBUG 2018/06/08 08:35:05 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvremove --autobackup=n -f /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_a0e0349eea524aa0f4d3ab0a34fa1304
>
>Result:   Logical volume "brick_a0e0349eea524aa0f4d3ab0a34fa1304" successfully removed
>[negroni] Started GET /queue/cc8c30c83cc8de532e15e823f1889908
>[negroni] Completed 200 OK in 269.434Âµs
>[kubeexec] DEBUG 2018/06/08 08:35:05 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvremove --autobackup=n -f /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_c91ff80db7a76c089815fbb5acd3b359
>
>Result:   Logical volume "brick_c91ff80db7a76c089815fbb5acd3b359" successfully removed
>[kubeexec] DEBUG 2018/06/08 08:35:06 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvremove --autobackup=n -f /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_50f7fa4e28c0ada62aee23b461d4a57b
>
>Result:   Logical volume "brick_50f7fa4e28c0ada62aee23b461d4a57b" successfully removed
>[kubeexec] DEBUG 2018/06/08 08:35:06 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvremove --autobackup=n -f /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_3ffb812f7f348d836df1be2db94be607
>
>Result:   Logical volume "brick_3ffb812f7f348d836df1be2db94be607" successfully removed
>[negroni] Started GET /queue/cc8c30c83cc8de532e15e823f1889908
>[negroni] Completed 200 OK in 154.638Âµs
>[kubeexec] DEBUG 2018/06/08 08:35:07 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvs --noheadings --options=thin_count   vg_3a4297677881963e3f80124971d50eea/tp_0fe2e9af9fd76f4fccdcf0e37a1f89ff
>
>Result:        0
>[kubeexec] DEBUG 2018/06/08 08:35:07 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvs --noheadings --options=thin_count   vg_9394bc70699b006c5460c9f654cf345f/tp_5bec7c663a12e79390ec9be885d88a1d
>
>Result:        0
>[negroni] Started GET /queue/cc8c30c83cc8de532e15e823f1889908
>[negroni] Completed 200 OK in 292.724Âµs
>[kubeexec] DEBUG 2018/06/08 08:35:07 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvs --noheadings --options=thin_count   vg_d389f0278a774bd7443a09af960961d8/tp_a0e0349eea524aa0f4d3ab0a34fa1304
>
>Result:        0
>[kubeexec] DEBUG 2018/06/08 08:35:08 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvs --noheadings --options=thin_count   vg_3a4297677881963e3f80124971d50eea/tp_c91ff80db7a76c089815fbb5acd3b359
>
>Result:        0
>[negroni] Started GET /queue/cc8c30c83cc8de532e15e823f1889908
>[negroni] Completed 200 OK in 213.515Âµs
>[kubeexec] DEBUG 2018/06/08 08:35:08 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvs --noheadings --options=thin_count   vg_96f1667f2f1ced2c5ef94772922be93b/tp_50f7fa4e28c0ada62aee23b461d4a57b
>
>Result:        0
>[kubeexec] DEBUG 2018/06/08 08:35:08 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvs --noheadings --options=thin_count   vg_b861af9cc67f5bf8d5a8f9e38f56b14e/tp_3ffb812f7f348d836df1be2db94be607
>
>Result:        0
>[negroni] Started GET /queue/cc8c30c83cc8de532e15e823f1889908
>[negroni] Completed 200 OK in 150.455Âµs
>[kubeexec] DEBUG 2018/06/08 08:35:09 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvremove --autobackup=n -f   vg_3a4297677881963e3f80124971d50eea/tp_0fe2e9af9fd76f4fccdcf0e37a1f89ff
>
>Result:   Logical volume "tp_0fe2e9af9fd76f4fccdcf0e37a1f89ff" successfully removed
>[kubeexec] DEBUG 2018/06/08 08:35:09 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvremove --autobackup=n -f   vg_9394bc70699b006c5460c9f654cf345f/tp_5bec7c663a12e79390ec9be885d88a1d
>
>Result:   Logical volume "tp_5bec7c663a12e79390ec9be885d88a1d" successfully removed
>[kubeexec] DEBUG 2018/06/08 08:35:09 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvremove --autobackup=n -f   vg_d389f0278a774bd7443a09af960961d8/tp_a0e0349eea524aa0f4d3ab0a34fa1304
>
>Result:   Logical volume "tp_a0e0349eea524aa0f4d3ab0a34fa1304" successfully removed
>[negroni] Started GET /queue/cc8c30c83cc8de532e15e823f1889908
>[negroni] Completed 200 OK in 164.743Âµs
>[kubeexec] DEBUG 2018/06/08 08:35:10 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvremove --autobackup=n -f   vg_3a4297677881963e3f80124971d50eea/tp_c91ff80db7a76c089815fbb5acd3b359
>
>Result:   Logical volume "tp_c91ff80db7a76c089815fbb5acd3b359" successfully removed
>[kubeexec] DEBUG 2018/06/08 08:35:10 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvremove --autobackup=n -f   vg_96f1667f2f1ced2c5ef94772922be93b/tp_50f7fa4e28c0ada62aee23b461d4a57b
>
>Result:   Logical volume "tp_50f7fa4e28c0ada62aee23b461d4a57b" successfully removed
>[kubeexec] DEBUG 2018/06/08 08:35:11 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvremove --autobackup=n -f   vg_b861af9cc67f5bf8d5a8f9e38f56b14e/tp_3ffb812f7f348d836df1be2db94be607
>
>Result:   Logical volume "tp_3ffb812f7f348d836df1be2db94be607" successfully removed
>[negroni] Started GET /queue/cc8c30c83cc8de532e15e823f1889908
>[negroni] Completed 200 OK in 144.584Âµs
>[kubeexec] DEBUG 2018/06/08 08:35:11 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: rmdir /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_0fe2e9af9fd76f4fccdcf0e37a1f89ff
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:35:12 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: rmdir /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_5bec7c663a12e79390ec9be885d88a1d
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:35:12 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: rmdir /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_a0e0349eea524aa0f4d3ab0a34fa1304
>Result: 
>[negroni] Started GET /queue/cc8c30c83cc8de532e15e823f1889908
>[negroni] Completed 200 OK in 256.528Âµs
>[kubeexec] DEBUG 2018/06/08 08:35:13 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: rmdir /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_c91ff80db7a76c089815fbb5acd3b359
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:35:13 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: rmdir /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_50f7fa4e28c0ada62aee23b461d4a57b
>Result: 
>[negroni] Started GET /queue/cc8c30c83cc8de532e15e823f1889908
>[negroni] Completed 200 OK in 214.385Âµs
>[kubeexec] DEBUG 2018/06/08 08:35:13 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: rmdir /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_3ffb812f7f348d836df1be2db94be607
>Result: 
>[heketi] INFO 2018/06/08 08:35:13 Delete Volume succeeded
>[asynchttp] INFO 2018/06/08 08:35:13 asynchttp.go:292: Completed job cc8c30c83cc8de532e15e823f1889908 in 20.129522656s
>[heketi] INFO 2018/06/08 08:35:14 Starting Node Health Status refresh
>[cmdexec] INFO 2018/06/08 08:35:14 Check Glusterd service status in node dhcp46-187.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 08:35:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:33 UTC; 2h 11min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48fed567_6ae4_11e8_bc4b_005056a5f18a.slice/docker-a506d9aaca4b440d184df40d01e90b2040811c8090bf4c19d892081307b28c0c.scope/system.slice/glusterd.service
>           ââ 433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ1228 /usr/sbin/glusterfsd -s 10.70.46.187 --volfile-id heketidbstorage.10.70.46.187.var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.187-var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.pid -S /var/run/gluster/c08898c2e3b0190c62df3a9d4a013c92.socket --brick-name /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.log --xlator-option *-posix.glusterd-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ9396 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/8fb729a0bbd51b204491c603c84acaa4.socket --xlator-option *replicate*.node-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034
>
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Unit glusterd.service entered failed state.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: glusterd.service failed.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:33 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 08:35:14 Periodic health check status: node 278bd6b4e16a8e62ef15aaae22e6abc1 up=true
>[cmdexec] INFO 2018/06/08 08:35:14 Check Glusterd service status in node dhcp46-122.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 08:35:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:42 UTC; 2h 11min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4900c968_6ae4_11e8_bc4b_005056a5f18a.slice/docker-438fb5fb16a82a3e701b28fa1f1ef8623dc37ca06374c604e5781a2b2ab2e06d.scope/system.slice/glusterd.service
>           ââ 433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ1106 /usr/sbin/glusterfsd -s 10.70.46.122 --volfile-id heketidbstorage.10.70.46.122.var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.122-var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.pid -S /var/run/gluster/9fa3ac3b12826912287adc88c26ac134.socket --brick-name /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_29e5b36d921cca1b0699871d9727c932/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.log --xlator-option *-posix.glusterd-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ9326 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f8994cdb09351d83e70da08261d99c64.socket --xlator-option *replicate*.node-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0
>
>Jun 08 06:23:40 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:42 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 08:35:14 Periodic health check status: node 70423cc0bcd044fe5ba8bbbd256a3e49 up=true
>[cmdexec] INFO 2018/06/08 08:35:14 Check Glusterd service status in node dhcp47-76.lab.eng.blr.redhat.com
>[negroni] Started GET /queue/cc8c30c83cc8de532e15e823f1889908
>[negroni] Completed 204 No Content in 142.056Âµs
>[kubeexec] DEBUG 2018/06/08 08:35:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 07:25:20 UTC; 1h 9min ago
>  Process: 828 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 830 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5cb2299_6aec_11e8_bc4b_005056a5f18a.slice/docker-7f9223abc9a8641cea68f374f4039235515978ee2dab970ac6cbef182c535426.scope/system.slice/glusterd.service
>           ââ 830 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ1197 /usr/sbin/glusterfsd -s 10.70.47.76 --volfile-id heketidbstorage.10.70.47.76.var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.76-var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.pid -S /var/run/gluster/b6976f9ddc3abc90e1dfa91c35a7c611.socket --brick-name /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_54c06cdf52c6fe1e3a3f8a9d39cde33d/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.log --xlator-option *-posix.glusterd-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ7514 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/ec468453b11caed5259f5e1abcc65b44.socket --xlator-option *replicate*.node-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94
>
>Jun 08 07:25:17 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 07:25:20 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 08:35:14 Periodic health check status: node d942fe6c0ee5691b7cc263968f97b650 up=true
>[heketi] INFO 2018/06/08 08:35:14 Cleaned 0 nodes from health cache
>[negroni] Started POST /volumes
>[heketi] INFO 2018/06/08 08:35:15 Allocating brick set #0
>[negroni] Completed 202 Accepted in 18.143951ms
>[asynchttp] INFO 2018/06/08 08:35:15 asynchttp.go:288: Started job c2357b5efc1fbcc9cef37b416ff6b7bc
>[heketi] INFO 2018/06/08 08:35:15 Started async operation: Create Volume
>[heketi] INFO 2018/06/08 08:35:15 Creating brick 924c9d4c901ad0decf1177966490ab2d
>[heketi] INFO 2018/06/08 08:35:15 Creating brick 0a21bd3dabbed98904d8c343a304aa58
>[heketi] INFO 2018/06/08 08:35:15 Creating brick 3075e0705c078c9e77f2395371953343
>[negroni] Started GET /queue/c2357b5efc1fbcc9cef37b416ff6b7bc
>[negroni] Completed 200 OK in 177.428Âµs
>[kubeexec] DEBUG 2018/06/08 08:35:15 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir -p /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_924c9d4c901ad0decf1177966490ab2d
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:35:15 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir -p /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_0a21bd3dabbed98904d8c343a304aa58
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:35:15 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvcreate --autobackup=n --poolmetadatasize 53248K --chunksize 256K --size 10485760K --thin vg_b861af9cc67f5bf8d5a8f9e38f56b14e/tp_924c9d4c901ad0decf1177966490ab2d --virtualsize 10485760K --name brick_924c9d4c901ad0decf1177966490ab2d
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_924c9d4c901ad0decf1177966490ab2d" created.
>[kubeexec] DEBUG 2018/06/08 08:35:15 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir -p /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_3075e0705c078c9e77f2395371953343
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:35:15 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_924c9d4c901ad0decf1177966490ab2d
>Result: meta-data=/dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_924c9d4c901ad0decf1177966490ab2d isize=512    agcount=16, agsize=163840 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=2621440, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:35:15 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: awk "BEGIN {print \"/dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_924c9d4c901ad0decf1177966490ab2d /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_924c9d4c901ad0decf1177966490ab2d xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:35:15 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvcreate --autobackup=n --poolmetadatasize 53248K --chunksize 256K --size 10485760K --thin vg_66a9af9f6bac95f9d8d556a2f14c29d3/tp_0a21bd3dabbed98904d8c343a304aa58 --virtualsize 10485760K --name brick_0a21bd3dabbed98904d8c343a304aa58
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_0a21bd3dabbed98904d8c343a304aa58" created.
>[kubeexec] DEBUG 2018/06/08 08:35:16 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_0a21bd3dabbed98904d8c343a304aa58
>Result: meta-data=/dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_0a21bd3dabbed98904d8c343a304aa58 isize=512    agcount=16, agsize=163840 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=2621440, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:35:16 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvcreate --autobackup=n --poolmetadatasize 53248K --chunksize 256K --size 10485760K --thin vg_96f1667f2f1ced2c5ef94772922be93b/tp_3075e0705c078c9e77f2395371953343 --virtualsize 10485760K --name brick_3075e0705c078c9e77f2395371953343
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_3075e0705c078c9e77f2395371953343" created.
>[kubeexec] DEBUG 2018/06/08 08:35:16 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_924c9d4c901ad0decf1177966490ab2d /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_924c9d4c901ad0decf1177966490ab2d
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:35:16 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: awk "BEGIN {print \"/dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_0a21bd3dabbed98904d8c343a304aa58 /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_0a21bd3dabbed98904d8c343a304aa58 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:35:16 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_924c9d4c901ad0decf1177966490ab2d/brick
>Result: 
>[negroni] Started GET /queue/c2357b5efc1fbcc9cef37b416ff6b7bc
>[negroni] Completed 200 OK in 127.317Âµs
>[kubeexec] DEBUG 2018/06/08 08:35:16 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_3075e0705c078c9e77f2395371953343
>Result: meta-data=/dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_3075e0705c078c9e77f2395371953343 isize=512    agcount=16, agsize=163840 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=2621440, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:35:16 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_0a21bd3dabbed98904d8c343a304aa58 /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_0a21bd3dabbed98904d8c343a304aa58
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:35:16 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: awk "BEGIN {print \"/dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_3075e0705c078c9e77f2395371953343 /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_3075e0705c078c9e77f2395371953343 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:35:16 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_0a21bd3dabbed98904d8c343a304aa58/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:35:16 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_3075e0705c078c9e77f2395371953343 /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_3075e0705c078c9e77f2395371953343
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:35:16 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_3075e0705c078c9e77f2395371953343/brick
>Result: 
>[cmdexec] INFO 2018/06/08 08:35:16 Creating volume vol_f08c8978937a9971fb637b34f333d318 replica 3
>[kubeexec] DEBUG 2018/06/08 08:35:16 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script volume create vol_f08c8978937a9971fb637b34f333d318 replica 3 10.70.46.122:/var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_0a21bd3dabbed98904d8c343a304aa58/brick 10.70.47.76:/var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_3075e0705c078c9e77f2395371953343/brick 10.70.46.187:/var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_924c9d4c901ad0decf1177966490ab2d/brick
>Result: volume create: vol_f08c8978937a9971fb637b34f333d318: success: please start the volume to access data
>[negroni] Started GET /queue/c2357b5efc1fbcc9cef37b416ff6b7bc
>[negroni] Completed 200 OK in 129.781Âµs
>[negroni] Started GET /queue/c2357b5efc1fbcc9cef37b416ff6b7bc
>[negroni] Completed 200 OK in 127.703Âµs
>[negroni] Started GET /queue/c2357b5efc1fbcc9cef37b416ff6b7bc
>[negroni] Completed 200 OK in 140.882Âµs
>[negroni] Started GET /queue/c2357b5efc1fbcc9cef37b416ff6b7bc
>[negroni] Completed 200 OK in 203.54Âµs
>[kubeexec] DEBUG 2018/06/08 08:35:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script volume start vol_f08c8978937a9971fb637b34f333d318
>Result: volume start: vol_f08c8978937a9971fb637b34f333d318: success
>[heketi] INFO 2018/06/08 08:35:21 Create Volume succeeded
>[asynchttp] INFO 2018/06/08 08:35:21 asynchttp.go:292: Completed job c2357b5efc1fbcc9cef37b416ff6b7bc in 5.808592133s
>[negroni] Started GET /queue/c2357b5efc1fbcc9cef37b416ff6b7bc
>[negroni] Completed 303 See Other in 234.518Âµs
>[negroni] Started GET /volumes/f08c8978937a9971fb637b34f333d318
>[negroni] Completed 200 OK in 3.374637ms
>[negroni] Started DELETE /volumes/f08c8978937a9971fb637b34f333d318
>[negroni] Completed 202 Accepted in 9.719209ms
>[asynchttp] INFO 2018/06/08 08:35:22 asynchttp.go:288: Started job 2b9ed3c45ef0a2c1d3e3e0fa5c33c832
>[heketi] INFO 2018/06/08 08:35:22 Started async operation: Delete Volume
>[negroni] Started GET /queue/2b9ed3c45ef0a2c1d3e3e0fa5c33c832
>[negroni] Completed 200 OK in 112.446Âµs
>[kubeexec] DEBUG 2018/06/08 08:35:22 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script snapshot list vol_f08c8978937a9971fb637b34f333d318 --xml
>Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
><cliOutput>
>  <opRet>0</opRet>
>  <opErrno>0</opErrno>
>  <opErrstr/>
>  <snapList>
>    <count>0</count>
>  </snapList>
></cliOutput>
>[negroni] Started GET /queue/2b9ed3c45ef0a2c1d3e3e0fa5c33c832
>[negroni] Completed 200 OK in 229.661Âµs
>[negroni] Started GET /queue/2b9ed3c45ef0a2c1d3e3e0fa5c33c832
>[negroni] Completed 200 OK in 229.361Âµs
>[kubeexec] DEBUG 2018/06/08 08:35:25 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script volume stop vol_f08c8978937a9971fb637b34f333d318 force
>Result: volume stop: vol_f08c8978937a9971fb637b34f333d318: success
>[kubeexec] DEBUG 2018/06/08 08:35:25 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script volume delete vol_f08c8978937a9971fb637b34f333d318
>Result: volume delete: vol_f08c8978937a9971fb637b34f333d318: success
>[heketi] INFO 2018/06/08 08:35:25 Deleting brick 0a21bd3dabbed98904d8c343a304aa58
>[heketi] INFO 2018/06/08 08:35:25 Deleting brick 3075e0705c078c9e77f2395371953343
>[heketi] INFO 2018/06/08 08:35:25 Deleting brick 924c9d4c901ad0decf1177966490ab2d
>[kubeexec] DEBUG 2018/06/08 08:35:25 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mount | grep -w /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_924c9d4c901ad0decf1177966490ab2d | cut -d" " -f1
>Result: /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_924c9d4c901ad0decf1177966490ab2d
>[kubeexec] DEBUG 2018/06/08 08:35:25 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mount | grep -w /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_0a21bd3dabbed98904d8c343a304aa58 | cut -d" " -f1
>Result: /dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_0a21bd3dabbed98904d8c343a304aa58
>[kubeexec] DEBUG 2018/06/08 08:35:25 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mount | grep -w /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_3075e0705c078c9e77f2395371953343 | cut -d" " -f1
>Result: /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_3075e0705c078c9e77f2395371953343
>[kubeexec] DEBUG 2018/06/08 08:35:25 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_0a21bd3dabbed98904d8c343a304aa58
>
>Result:   vg_66a9af9f6bac95f9d8d556a2f14c29d3/tp_0a21bd3dabbed98904d8c343a304aa58
>[kubeexec] DEBUG 2018/06/08 08:35:25 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_924c9d4c901ad0decf1177966490ab2d
>
>Result:   vg_b861af9cc67f5bf8d5a8f9e38f56b14e/tp_924c9d4c901ad0decf1177966490ab2d
>[negroni] Started GET /queue/2b9ed3c45ef0a2c1d3e3e0fa5c33c832
>[negroni] Completed 200 OK in 208.912Âµs
>[kubeexec] DEBUG 2018/06/08 08:35:26 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_3075e0705c078c9e77f2395371953343
>
>Result:   vg_96f1667f2f1ced2c5ef94772922be93b/tp_3075e0705c078c9e77f2395371953343
>[kubeexec] DEBUG 2018/06/08 08:35:26 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: umount /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_0a21bd3dabbed98904d8c343a304aa58
>Result: 
>[negroni] Started GET /queue/2b9ed3c45ef0a2c1d3e3e0fa5c33c832
>[negroni] Completed 200 OK in 170.333Âµs
>[kubeexec] DEBUG 2018/06/08 08:35:26 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: umount /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_924c9d4c901ad0decf1177966490ab2d
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:35:27 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: umount /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_3075e0705c078c9e77f2395371953343
>Result: 
>[negroni] Started GET /queue/2b9ed3c45ef0a2c1d3e3e0fa5c33c832
>[negroni] Completed 200 OK in 180.986Âµs
>[kubeexec] DEBUG 2018/06/08 08:35:27 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: sed -i.save "/brick_0a21bd3dabbed98904d8c343a304aa58/d" /var/lib/heketi/fstab
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:35:27 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: sed -i.save "/brick_924c9d4c901ad0decf1177966490ab2d/d" /var/lib/heketi/fstab
>Result: 
>[negroni] Started GET /queue/2b9ed3c45ef0a2c1d3e3e0fa5c33c832
>[negroni] Completed 200 OK in 229.308Âµs
>[kubeexec] DEBUG 2018/06/08 08:35:28 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: sed -i.save "/brick_3075e0705c078c9e77f2395371953343/d" /var/lib/heketi/fstab
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:35:28 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvremove --autobackup=n -f /dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_0a21bd3dabbed98904d8c343a304aa58
>
>Result:   Logical volume "brick_0a21bd3dabbed98904d8c343a304aa58" successfully removed
>[kubeexec] DEBUG 2018/06/08 08:35:29 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvremove --autobackup=n -f /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_924c9d4c901ad0decf1177966490ab2d
>
>Result:   Logical volume "brick_924c9d4c901ad0decf1177966490ab2d" successfully removed
>[negroni] Started GET /queue/2b9ed3c45ef0a2c1d3e3e0fa5c33c832
>[negroni] Completed 200 OK in 214.255Âµs
>[kubeexec] DEBUG 2018/06/08 08:35:29 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvremove --autobackup=n -f /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_3075e0705c078c9e77f2395371953343
>
>Result:   Logical volume "brick_3075e0705c078c9e77f2395371953343" successfully removed
>[kubeexec] DEBUG 2018/06/08 08:35:30 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvs --noheadings --options=thin_count   vg_66a9af9f6bac95f9d8d556a2f14c29d3/tp_0a21bd3dabbed98904d8c343a304aa58
>
>Result:        0
>[kubeexec] DEBUG 2018/06/08 08:35:30 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvs --noheadings --options=thin_count   vg_b861af9cc67f5bf8d5a8f9e38f56b14e/tp_924c9d4c901ad0decf1177966490ab2d
>
>Result:        0
>[negroni] Started GET /queue/2b9ed3c45ef0a2c1d3e3e0fa5c33c832
>[negroni] Completed 200 OK in 163.729Âµs
>[kubeexec] DEBUG 2018/06/08 08:35:31 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvs --noheadings --options=thin_count   vg_96f1667f2f1ced2c5ef94772922be93b/tp_3075e0705c078c9e77f2395371953343
>
>Result:        0
>[kubeexec] DEBUG 2018/06/08 08:35:31 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvremove --autobackup=n -f   vg_66a9af9f6bac95f9d8d556a2f14c29d3/tp_0a21bd3dabbed98904d8c343a304aa58
>
>Result:   Logical volume "tp_0a21bd3dabbed98904d8c343a304aa58" successfully removed
>[kubeexec] DEBUG 2018/06/08 08:35:31 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvremove --autobackup=n -f   vg_b861af9cc67f5bf8d5a8f9e38f56b14e/tp_924c9d4c901ad0decf1177966490ab2d
>
>Result:   Logical volume "tp_924c9d4c901ad0decf1177966490ab2d" successfully removed
>[negroni] Started GET /queue/2b9ed3c45ef0a2c1d3e3e0fa5c33c832
>[negroni] Completed 200 OK in 141.089Âµs
>[kubeexec] DEBUG 2018/06/08 08:35:32 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvremove --autobackup=n -f   vg_96f1667f2f1ced2c5ef94772922be93b/tp_3075e0705c078c9e77f2395371953343
>
>Result:   Logical volume "tp_3075e0705c078c9e77f2395371953343" successfully removed
>[kubeexec] DEBUG 2018/06/08 08:35:32 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: rmdir /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_0a21bd3dabbed98904d8c343a304aa58
>Result: 
>[negroni] Started GET /queue/2b9ed3c45ef0a2c1d3e3e0fa5c33c832
>[negroni] Completed 200 OK in 174.937Âµs
>[kubeexec] DEBUG 2018/06/08 08:35:32 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: rmdir /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_924c9d4c901ad0decf1177966490ab2d
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:35:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: rmdir /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_3075e0705c078c9e77f2395371953343
>Result: 
>[heketi] INFO 2018/06/08 08:35:33 Delete Volume succeeded
>[asynchttp] INFO 2018/06/08 08:35:33 asynchttp.go:292: Completed job 2b9ed3c45ef0a2c1d3e3e0fa5c33c832 in 10.544537747s
>[negroni] Started GET /queue/2b9ed3c45ef0a2c1d3e3e0fa5c33c832
>[negroni] Completed 204 No Content in 167.211Âµs
>[negroni] Started POST /volumes
>[heketi] INFO 2018/06/08 08:35:33 Allocating brick set #0
>[heketi] INFO 2018/06/08 08:35:33 Allocating brick set #0
>[heketi] INFO 2018/06/08 08:35:33 Allocating brick set #1
>[heketi] INFO 2018/06/08 08:35:33 Allocating brick set #0
>[heketi] INFO 2018/06/08 08:35:33 Allocating brick set #1
>[heketi] INFO 2018/06/08 08:35:33 Allocating brick set #2
>[heketi] INFO 2018/06/08 08:35:33 Allocating brick set #3
>[heketi] INFO 2018/06/08 08:35:33 Allocating brick set #0
>[heketi] INFO 2018/06/08 08:35:33 Allocating brick set #1
>[heketi] INFO 2018/06/08 08:35:33 Allocating brick set #2
>[heketi] INFO 2018/06/08 08:35:33 Allocating brick set #3
>[heketi] INFO 2018/06/08 08:35:33 Allocating brick set #4
>[heketi] INFO 2018/06/08 08:35:33 Allocating brick set #5
>[heketi] INFO 2018/06/08 08:35:33 Allocating brick set #6
>[heketi] INFO 2018/06/08 08:35:33 Allocating brick set #7
>[heketi] ERROR 2018/06/08 08:35:33 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:1167: Create Volume Build Failed: No space
>[negroni] Completed 500 Internal Server Error in 21.13287ms
>[negroni] Started POST /volumes
>[heketi] INFO 2018/06/08 08:35:34 Allocating brick set #0
>[negroni] Completed 202 Accepted in 11.697203ms
>[asynchttp] INFO 2018/06/08 08:35:34 asynchttp.go:288: Started job 70dc6ad014f42c11e49962509fcef1b7
>[heketi] INFO 2018/06/08 08:35:34 Started async operation: Create Volume
>[heketi] INFO 2018/06/08 08:35:34 Creating brick 6a50e9b2961a5cd8deafffc0f1e7d401
>[heketi] INFO 2018/06/08 08:35:34 Creating brick 42b57e50453bebac4a1d2fdcd70c3418
>[heketi] INFO 2018/06/08 08:35:34 Creating brick 3b99cfab90966e2c5ccb9ef5ab1cfa1d
>[negroni] Started GET /queue/70dc6ad014f42c11e49962509fcef1b7
>[negroni] Completed 200 OK in 98.431Âµs
>[kubeexec] DEBUG 2018/06/08 08:35:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir -p /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_6a50e9b2961a5cd8deafffc0f1e7d401
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:35:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir -p /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_3b99cfab90966e2c5ccb9ef5ab1cfa1d
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:35:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir -p /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_42b57e50453bebac4a1d2fdcd70c3418
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:35:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvcreate --autobackup=n --poolmetadatasize 53248K --chunksize 256K --size 10485760K --thin vg_66a9af9f6bac95f9d8d556a2f14c29d3/tp_6a50e9b2961a5cd8deafffc0f1e7d401 --virtualsize 10485760K --name brick_6a50e9b2961a5cd8deafffc0f1e7d401
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_6a50e9b2961a5cd8deafffc0f1e7d401" created.
>[kubeexec] DEBUG 2018/06/08 08:35:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvcreate --autobackup=n --poolmetadatasize 53248K --chunksize 256K --size 10485760K --thin vg_b861af9cc67f5bf8d5a8f9e38f56b14e/tp_3b99cfab90966e2c5ccb9ef5ab1cfa1d --virtualsize 10485760K --name brick_3b99cfab90966e2c5ccb9ef5ab1cfa1d
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_3b99cfab90966e2c5ccb9ef5ab1cfa1d" created.
>[kubeexec] DEBUG 2018/06/08 08:35:35 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvcreate --autobackup=n --poolmetadatasize 53248K --chunksize 256K --size 10485760K --thin vg_96f1667f2f1ced2c5ef94772922be93b/tp_42b57e50453bebac4a1d2fdcd70c3418 --virtualsize 10485760K --name brick_42b57e50453bebac4a1d2fdcd70c3418
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_42b57e50453bebac4a1d2fdcd70c3418" created.
>[kubeexec] DEBUG 2018/06/08 08:35:35 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_6a50e9b2961a5cd8deafffc0f1e7d401
>Result: meta-data=/dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_6a50e9b2961a5cd8deafffc0f1e7d401 isize=512    agcount=16, agsize=163840 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=2621440, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:35:35 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_3b99cfab90966e2c5ccb9ef5ab1cfa1d
>Result: meta-data=/dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_3b99cfab90966e2c5ccb9ef5ab1cfa1d isize=512    agcount=16, agsize=163840 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=2621440, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:35:35 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_42b57e50453bebac4a1d2fdcd70c3418
>Result: meta-data=/dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_42b57e50453bebac4a1d2fdcd70c3418 isize=512    agcount=16, agsize=163840 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=2621440, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:35:35 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: awk "BEGIN {print \"/dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_6a50e9b2961a5cd8deafffc0f1e7d401 /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_6a50e9b2961a5cd8deafffc0f1e7d401 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:35:35 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: awk "BEGIN {print \"/dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_3b99cfab90966e2c5ccb9ef5ab1cfa1d /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_3b99cfab90966e2c5ccb9ef5ab1cfa1d xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:35:35 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: awk "BEGIN {print \"/dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_42b57e50453bebac4a1d2fdcd70c3418 /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_42b57e50453bebac4a1d2fdcd70c3418 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:35:35 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_3b99cfab90966e2c5ccb9ef5ab1cfa1d /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_3b99cfab90966e2c5ccb9ef5ab1cfa1d
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:35:35 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_6a50e9b2961a5cd8deafffc0f1e7d401 /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_6a50e9b2961a5cd8deafffc0f1e7d401
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:35:35 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_42b57e50453bebac4a1d2fdcd70c3418 /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_42b57e50453bebac4a1d2fdcd70c3418
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:35:35 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_3b99cfab90966e2c5ccb9ef5ab1cfa1d/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:35:35 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_6a50e9b2961a5cd8deafffc0f1e7d401/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:35:35 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_42b57e50453bebac4a1d2fdcd70c3418/brick
>Result: 
>[cmdexec] INFO 2018/06/08 08:35:35 Creating volume vol_4a5dacb31fced8db04ae8887e9b873ac replica 3
>[negroni] Started GET /queue/70dc6ad014f42c11e49962509fcef1b7
>[negroni] Completed 200 OK in 139.572Âµs
>[kubeexec] DEBUG 2018/06/08 08:35:35 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script volume create vol_4a5dacb31fced8db04ae8887e9b873ac replica 3 10.70.46.122:/var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_6a50e9b2961a5cd8deafffc0f1e7d401/brick 10.70.47.76:/var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_42b57e50453bebac4a1d2fdcd70c3418/brick 10.70.46.187:/var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_3b99cfab90966e2c5ccb9ef5ab1cfa1d/brick
>Result: volume create: vol_4a5dacb31fced8db04ae8887e9b873ac: success: please start the volume to access data
>[negroni] Started GET /queue/70dc6ad014f42c11e49962509fcef1b7
>[negroni] Completed 200 OK in 141.181Âµs
>[negroni] Started GET /queue/70dc6ad014f42c11e49962509fcef1b7
>[negroni] Completed 200 OK in 140.438Âµs
>[kubeexec] DEBUG 2018/06/08 08:35:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script volume start vol_4a5dacb31fced8db04ae8887e9b873ac
>Result: volume start: vol_4a5dacb31fced8db04ae8887e9b873ac: success
>[heketi] INFO 2018/06/08 08:35:37 Create Volume succeeded
>[asynchttp] INFO 2018/06/08 08:35:37 asynchttp.go:292: Completed job 70dc6ad014f42c11e49962509fcef1b7 in 3.457453444s
>[negroni] Started GET /queue/70dc6ad014f42c11e49962509fcef1b7
>[negroni] Completed 303 See Other in 213.513Âµs
>[negroni] Started GET /volumes/4a5dacb31fced8db04ae8887e9b873ac
>[negroni] Completed 200 OK in 3.291239ms
>[negroni] Started GET /clusters
>[negroni] Completed 200 OK in 146.999Âµs
>[negroni] Started GET /clusters/0a73c60efdd4673113b668afea101e6d
>[negroni] Completed 200 OK in 315.345Âµs
>[negroni] Started GET /nodes/278bd6b4e16a8e62ef15aaae22e6abc1
>[negroni] Completed 200 OK in 1.563342ms
>[negroni] Started GET /nodes/70423cc0bcd044fe5ba8bbbd256a3e49
>[negroni] Completed 200 OK in 941.912Âµs
>[negroni] Started GET /nodes/d942fe6c0ee5691b7cc263968f97b650
>[negroni] Completed 200 OK in 1.424225ms
>[negroni] Started DELETE /volumes/4a5dacb31fced8db04ae8887e9b873ac
>[negroni] Completed 202 Accepted in 11.916142ms
>[asynchttp] INFO 2018/06/08 08:35:39 asynchttp.go:288: Started job 43057859a006c8ad74e7f5bf974ebf21
>[heketi] INFO 2018/06/08 08:35:39 Started async operation: Delete Volume
>[negroni] Started GET /queue/43057859a006c8ad74e7f5bf974ebf21
>[negroni] Completed 200 OK in 137.452Âµs
>[kubeexec] DEBUG 2018/06/08 08:35:39 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: gluster --mode=script snapshot list vol_4a5dacb31fced8db04ae8887e9b873ac --xml
>Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
><cliOutput>
>  <opRet>0</opRet>
>  <opErrno>0</opErrno>
>  <opErrstr/>
>  <snapList>
>    <count>0</count>
>  </snapList>
></cliOutput>
>[negroni] Started GET /queue/43057859a006c8ad74e7f5bf974ebf21
>[negroni] Completed 200 OK in 165.247Âµs
>[negroni] Started GET /queue/43057859a006c8ad74e7f5bf974ebf21
>[negroni] Completed 200 OK in 135.016Âµs
>[negroni] Started GET /queue/43057859a006c8ad74e7f5bf974ebf21
>[negroni] Completed 200 OK in 210.597Âµs
>[kubeexec] DEBUG 2018/06/08 08:35:42 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: gluster --mode=script volume stop vol_4a5dacb31fced8db04ae8887e9b873ac force
>Result: volume stop: vol_4a5dacb31fced8db04ae8887e9b873ac: success
>[kubeexec] DEBUG 2018/06/08 08:35:42 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: gluster --mode=script volume delete vol_4a5dacb31fced8db04ae8887e9b873ac
>Result: volume delete: vol_4a5dacb31fced8db04ae8887e9b873ac: success
>[heketi] INFO 2018/06/08 08:35:42 Deleting brick 42b57e50453bebac4a1d2fdcd70c3418
>[heketi] INFO 2018/06/08 08:35:42 Deleting brick 3b99cfab90966e2c5ccb9ef5ab1cfa1d
>[heketi] INFO 2018/06/08 08:35:42 Deleting brick 6a50e9b2961a5cd8deafffc0f1e7d401
>[kubeexec] DEBUG 2018/06/08 08:35:42 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mount | grep -w /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_6a50e9b2961a5cd8deafffc0f1e7d401 | cut -d" " -f1
>Result: /dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_6a50e9b2961a5cd8deafffc0f1e7d401
>[kubeexec] DEBUG 2018/06/08 08:35:42 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mount | grep -w /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_42b57e50453bebac4a1d2fdcd70c3418 | cut -d" " -f1
>Result: /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_42b57e50453bebac4a1d2fdcd70c3418
>[kubeexec] DEBUG 2018/06/08 08:35:42 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mount | grep -w /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_3b99cfab90966e2c5ccb9ef5ab1cfa1d | cut -d" " -f1
>Result: /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_3b99cfab90966e2c5ccb9ef5ab1cfa1d
>[kubeexec] DEBUG 2018/06/08 08:35:43 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_6a50e9b2961a5cd8deafffc0f1e7d401
>
>Result:   vg_66a9af9f6bac95f9d8d556a2f14c29d3/tp_6a50e9b2961a5cd8deafffc0f1e7d401
>[negroni] Started GET /queue/43057859a006c8ad74e7f5bf974ebf21
>[negroni] Completed 200 OK in 138.076Âµs
>[kubeexec] DEBUG 2018/06/08 08:35:43 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_42b57e50453bebac4a1d2fdcd70c3418
>
>Result:   vg_96f1667f2f1ced2c5ef94772922be93b/tp_42b57e50453bebac4a1d2fdcd70c3418
>[kubeexec] DEBUG 2018/06/08 08:35:43 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_3b99cfab90966e2c5ccb9ef5ab1cfa1d
>
>Result:   vg_b861af9cc67f5bf8d5a8f9e38f56b14e/tp_3b99cfab90966e2c5ccb9ef5ab1cfa1d
>[negroni] Started GET /queue/43057859a006c8ad74e7f5bf974ebf21
>[negroni] Completed 200 OK in 113.015Âµs
>[kubeexec] DEBUG 2018/06/08 08:35:44 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: umount /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_6a50e9b2961a5cd8deafffc0f1e7d401
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:35:44 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: umount /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_42b57e50453bebac4a1d2fdcd70c3418
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:35:44 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: umount /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_3b99cfab90966e2c5ccb9ef5ab1cfa1d
>Result: 
>[negroni] Started GET /queue/43057859a006c8ad74e7f5bf974ebf21
>[negroni] Completed 200 OK in 147.562Âµs
>[kubeexec] DEBUG 2018/06/08 08:35:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: sed -i.save "/brick_6a50e9b2961a5cd8deafffc0f1e7d401/d" /var/lib/heketi/fstab
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:35:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: sed -i.save "/brick_42b57e50453bebac4a1d2fdcd70c3418/d" /var/lib/heketi/fstab
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:35:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: sed -i.save "/brick_3b99cfab90966e2c5ccb9ef5ab1cfa1d/d" /var/lib/heketi/fstab
>Result: 
>[negroni] Started GET /queue/43057859a006c8ad74e7f5bf974ebf21
>[negroni] Completed 200 OK in 158.668Âµs
>[kubeexec] DEBUG 2018/06/08 08:35:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvremove --autobackup=n -f /dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_6a50e9b2961a5cd8deafffc0f1e7d401
>
>Result:   Logical volume "brick_6a50e9b2961a5cd8deafffc0f1e7d401" successfully removed
>[kubeexec] DEBUG 2018/06/08 08:35:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvremove --autobackup=n -f /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_42b57e50453bebac4a1d2fdcd70c3418
>
>Result:   Logical volume "brick_42b57e50453bebac4a1d2fdcd70c3418" successfully removed
>[kubeexec] DEBUG 2018/06/08 08:35:47 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvremove --autobackup=n -f /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_3b99cfab90966e2c5ccb9ef5ab1cfa1d
>
>Result:   Logical volume "brick_3b99cfab90966e2c5ccb9ef5ab1cfa1d" successfully removed
>[negroni] Started GET /queue/43057859a006c8ad74e7f5bf974ebf21
>[negroni] Completed 200 OK in 180.712Âµs
>[kubeexec] DEBUG 2018/06/08 08:35:47 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvs --noheadings --options=thin_count   vg_66a9af9f6bac95f9d8d556a2f14c29d3/tp_6a50e9b2961a5cd8deafffc0f1e7d401
>
>Result:        0
>[kubeexec] DEBUG 2018/06/08 08:35:48 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvs --noheadings --options=thin_count   vg_96f1667f2f1ced2c5ef94772922be93b/tp_42b57e50453bebac4a1d2fdcd70c3418
>
>Result:        0
>[negroni] Started GET /queue/43057859a006c8ad74e7f5bf974ebf21
>[negroni] Completed 200 OK in 204.419Âµs
>[kubeexec] DEBUG 2018/06/08 08:35:48 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvs --noheadings --options=thin_count   vg_b861af9cc67f5bf8d5a8f9e38f56b14e/tp_3b99cfab90966e2c5ccb9ef5ab1cfa1d
>
>Result:        0
>[kubeexec] DEBUG 2018/06/08 08:35:49 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvremove --autobackup=n -f   vg_66a9af9f6bac95f9d8d556a2f14c29d3/tp_6a50e9b2961a5cd8deafffc0f1e7d401
>
>Result:   Logical volume "tp_6a50e9b2961a5cd8deafffc0f1e7d401" successfully removed
>[negroni] Started GET /queue/43057859a006c8ad74e7f5bf974ebf21
>[negroni] Completed 200 OK in 218.835Âµs
>[kubeexec] DEBUG 2018/06/08 08:35:49 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvremove --autobackup=n -f   vg_96f1667f2f1ced2c5ef94772922be93b/tp_42b57e50453bebac4a1d2fdcd70c3418
>
>Result:   Logical volume "tp_42b57e50453bebac4a1d2fdcd70c3418" successfully removed
>[kubeexec] DEBUG 2018/06/08 08:35:49 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvremove --autobackup=n -f   vg_b861af9cc67f5bf8d5a8f9e38f56b14e/tp_3b99cfab90966e2c5ccb9ef5ab1cfa1d
>
>Result:   Logical volume "tp_3b99cfab90966e2c5ccb9ef5ab1cfa1d" successfully removed
>[negroni] Started GET /queue/43057859a006c8ad74e7f5bf974ebf21
>[negroni] Completed 200 OK in 220.978Âµs
>[kubeexec] DEBUG 2018/06/08 08:35:50 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: rmdir /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_6a50e9b2961a5cd8deafffc0f1e7d401
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:35:50 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: rmdir /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_42b57e50453bebac4a1d2fdcd70c3418
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:35:50 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: rmdir /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_3b99cfab90966e2c5ccb9ef5ab1cfa1d
>Result: 
>[heketi] INFO 2018/06/08 08:35:50 Delete Volume succeeded
>[asynchttp] INFO 2018/06/08 08:35:50 asynchttp.go:292: Completed job 43057859a006c8ad74e7f5bf974ebf21 in 11.548980199s
>[negroni] Started GET /queue/43057859a006c8ad74e7f5bf974ebf21
>[negroni] Completed 204 No Content in 196.151Âµs
>[negroni] Started GET /clusters
>[negroni] Completed 200 OK in 2.250353ms
>[negroni] Started GET /clusters/0a73c60efdd4673113b668afea101e6d
>[negroni] Completed 200 OK in 372.524Âµs
>[negroni] Started GET /nodes/278bd6b4e16a8e62ef15aaae22e6abc1
>[negroni] Completed 200 OK in 1.364688ms
>[negroni] Started GET /nodes/70423cc0bcd044fe5ba8bbbd256a3e49
>[negroni] Completed 200 OK in 1.494786ms
>[negroni] Started GET /nodes/d942fe6c0ee5691b7cc263968f97b650
>[negroni] Completed 200 OK in 1.483396ms
>[negroni] Started POST /volumes
>[heketi] INFO 2018/06/08 08:35:51 Allocating brick set #0
>[negroni] Completed 202 Accepted in 12.952701ms
>[asynchttp] INFO 2018/06/08 08:35:51 asynchttp.go:288: Started job a9ac68956dfb4c06cf5ad2dc5bbcaae8
>[heketi] INFO 2018/06/08 08:35:51 Started async operation: Create Volume
>[heketi] INFO 2018/06/08 08:35:51 Creating brick 40cd1137d297de071f2da2a433be6fd7
>[heketi] INFO 2018/06/08 08:35:51 Creating brick 89391ff92f83c755ffb9c270e0f6f3c9
>[heketi] INFO 2018/06/08 08:35:51 Creating brick 16e82f0f7d6e040618ce84658b250ceb
>[negroni] Started GET /queue/a9ac68956dfb4c06cf5ad2dc5bbcaae8
>[negroni] Completed 200 OK in 102.487Âµs
>[kubeexec] DEBUG 2018/06/08 08:35:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir -p /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_89391ff92f83c755ffb9c270e0f6f3c9
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:35:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir -p /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_40cd1137d297de071f2da2a433be6fd7
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:35:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir -p /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_16e82f0f7d6e040618ce84658b250ceb
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:35:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvcreate --autobackup=n --poolmetadatasize 53248K --chunksize 256K --size 10485760K --thin vg_b861af9cc67f5bf8d5a8f9e38f56b14e/tp_89391ff92f83c755ffb9c270e0f6f3c9 --virtualsize 10485760K --name brick_89391ff92f83c755ffb9c270e0f6f3c9
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_89391ff92f83c755ffb9c270e0f6f3c9" created.
>[kubeexec] DEBUG 2018/06/08 08:35:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvcreate --autobackup=n --poolmetadatasize 53248K --chunksize 256K --size 10485760K --thin vg_66a9af9f6bac95f9d8d556a2f14c29d3/tp_40cd1137d297de071f2da2a433be6fd7 --virtualsize 10485760K --name brick_40cd1137d297de071f2da2a433be6fd7
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_40cd1137d297de071f2da2a433be6fd7" created.
>[kubeexec] DEBUG 2018/06/08 08:35:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvcreate --autobackup=n --poolmetadatasize 53248K --chunksize 256K --size 10485760K --thin vg_96f1667f2f1ced2c5ef94772922be93b/tp_16e82f0f7d6e040618ce84658b250ceb --virtualsize 10485760K --name brick_16e82f0f7d6e040618ce84658b250ceb
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_16e82f0f7d6e040618ce84658b250ceb" created.
>[kubeexec] DEBUG 2018/06/08 08:35:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_89391ff92f83c755ffb9c270e0f6f3c9
>Result: meta-data=/dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_89391ff92f83c755ffb9c270e0f6f3c9 isize=512    agcount=16, agsize=163840 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=2621440, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:35:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_40cd1137d297de071f2da2a433be6fd7
>Result: meta-data=/dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_40cd1137d297de071f2da2a433be6fd7 isize=512    agcount=16, agsize=163840 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=2621440, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:35:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_16e82f0f7d6e040618ce84658b250ceb
>Result: meta-data=/dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_16e82f0f7d6e040618ce84658b250ceb isize=512    agcount=16, agsize=163840 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=2621440, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:35:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: awk "BEGIN {print \"/dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_89391ff92f83c755ffb9c270e0f6f3c9 /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_89391ff92f83c755ffb9c270e0f6f3c9 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:35:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: awk "BEGIN {print \"/dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_40cd1137d297de071f2da2a433be6fd7 /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_40cd1137d297de071f2da2a433be6fd7 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:35:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: awk "BEGIN {print \"/dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_16e82f0f7d6e040618ce84658b250ceb /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_16e82f0f7d6e040618ce84658b250ceb xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:35:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_89391ff92f83c755ffb9c270e0f6f3c9 /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_89391ff92f83c755ffb9c270e0f6f3c9
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:35:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_40cd1137d297de071f2da2a433be6fd7 /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_40cd1137d297de071f2da2a433be6fd7
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:35:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_16e82f0f7d6e040618ce84658b250ceb /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_16e82f0f7d6e040618ce84658b250ceb
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:35:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_89391ff92f83c755ffb9c270e0f6f3c9/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:35:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_16e82f0f7d6e040618ce84658b250ceb/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:35:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_40cd1137d297de071f2da2a433be6fd7/brick
>Result: 
>[cmdexec] INFO 2018/06/08 08:35:52 Creating volume vol_27b4e68c05a8cded57c197df819b2c12 replica 3
>[negroni] Started GET /queue/a9ac68956dfb4c06cf5ad2dc5bbcaae8
>[negroni] Completed 200 OK in 138.108Âµs
>[kubeexec] DEBUG 2018/06/08 08:35:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script volume create vol_27b4e68c05a8cded57c197df819b2c12 replica 3 10.70.46.122:/var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_40cd1137d297de071f2da2a433be6fd7/brick 10.70.47.76:/var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_16e82f0f7d6e040618ce84658b250ceb/brick 10.70.46.187:/var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_89391ff92f83c755ffb9c270e0f6f3c9/brick
>Result: volume create: vol_27b4e68c05a8cded57c197df819b2c12: success: please start the volume to access data
>[negroni] Started GET /queue/a9ac68956dfb4c06cf5ad2dc5bbcaae8
>[negroni] Completed 200 OK in 136.92Âµs
>[negroni] Started GET /queue/a9ac68956dfb4c06cf5ad2dc5bbcaae8
>[negroni] Completed 200 OK in 132.062Âµs
>[negroni] Started GET /queue/a9ac68956dfb4c06cf5ad2dc5bbcaae8
>[negroni] Completed 200 OK in 147.524Âµs
>[negroni] Started GET /queue/a9ac68956dfb4c06cf5ad2dc5bbcaae8
>[negroni] Completed 200 OK in 324.84Âµs
>[kubeexec] DEBUG 2018/06/08 08:35:57 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script volume start vol_27b4e68c05a8cded57c197df819b2c12
>Result: volume start: vol_27b4e68c05a8cded57c197df819b2c12: success
>[heketi] INFO 2018/06/08 08:35:57 Create Volume succeeded
>[asynchttp] INFO 2018/06/08 08:35:57 asynchttp.go:292: Completed job a9ac68956dfb4c06cf5ad2dc5bbcaae8 in 5.417938255s
>[negroni] Started GET /queue/a9ac68956dfb4c06cf5ad2dc5bbcaae8
>[negroni] Completed 303 See Other in 201.089Âµs
>[negroni] Started GET /volumes/27b4e68c05a8cded57c197df819b2c12
>[negroni] Completed 200 OK in 3.059591ms
>[negroni] Started POST /volumes
>[heketi] INFO 2018/06/08 08:35:57 Allocating brick set #0
>[negroni] Completed 202 Accepted in 15.237599ms
>[asynchttp] INFO 2018/06/08 08:35:57 asynchttp.go:288: Started job 138190be1330787bd37eed0a371f160f
>[heketi] INFO 2018/06/08 08:35:57 Started async operation: Create Volume
>[heketi] INFO 2018/06/08 08:35:57 Creating brick b5d1434de12b984177d8fe3865f88829
>[heketi] INFO 2018/06/08 08:35:57 Creating brick 3f6e5bae8b9cfcddcc7d0f9b8ded99fc
>[heketi] INFO 2018/06/08 08:35:57 Creating brick cde06dca54580b41329bf54d8306df4c
>[negroni] Started GET /queue/138190be1330787bd37eed0a371f160f
>[negroni] Completed 200 OK in 315.206Âµs
>[kubeexec] DEBUG 2018/06/08 08:35:58 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir -p /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_cde06dca54580b41329bf54d8306df4c
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:35:58 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir -p /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_3f6e5bae8b9cfcddcc7d0f9b8ded99fc
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:35:58 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir -p /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_b5d1434de12b984177d8fe3865f88829
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:35:58 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvcreate --autobackup=n --poolmetadatasize 53248K --chunksize 256K --size 10485760K --thin vg_3a4297677881963e3f80124971d50eea/tp_3f6e5bae8b9cfcddcc7d0f9b8ded99fc --virtualsize 10485760K --name brick_3f6e5bae8b9cfcddcc7d0f9b8ded99fc
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_3f6e5bae8b9cfcddcc7d0f9b8ded99fc" created.
>[kubeexec] DEBUG 2018/06/08 08:35:58 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvcreate --autobackup=n --poolmetadatasize 53248K --chunksize 256K --size 10485760K --thin vg_9394bc70699b006c5460c9f654cf345f/tp_b5d1434de12b984177d8fe3865f88829 --virtualsize 10485760K --name brick_b5d1434de12b984177d8fe3865f88829
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_b5d1434de12b984177d8fe3865f88829" created.
>[kubeexec] DEBUG 2018/06/08 08:35:58 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvcreate --autobackup=n --poolmetadatasize 53248K --chunksize 256K --size 10485760K --thin vg_d389f0278a774bd7443a09af960961d8/tp_cde06dca54580b41329bf54d8306df4c --virtualsize 10485760K --name brick_cde06dca54580b41329bf54d8306df4c
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_cde06dca54580b41329bf54d8306df4c" created.
>[kubeexec] DEBUG 2018/06/08 08:35:58 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_3f6e5bae8b9cfcddcc7d0f9b8ded99fc
>Result: meta-data=/dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_3f6e5bae8b9cfcddcc7d0f9b8ded99fc isize=512    agcount=16, agsize=163840 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=2621440, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:35:58 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_b5d1434de12b984177d8fe3865f88829
>Result: meta-data=/dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_b5d1434de12b984177d8fe3865f88829 isize=512    agcount=16, agsize=163840 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=2621440, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:35:58 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_cde06dca54580b41329bf54d8306df4c
>Result: meta-data=/dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_cde06dca54580b41329bf54d8306df4c isize=512    agcount=16, agsize=163840 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=2621440, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:35:58 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: awk "BEGIN {print \"/dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_3f6e5bae8b9cfcddcc7d0f9b8ded99fc /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_3f6e5bae8b9cfcddcc7d0f9b8ded99fc xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:35:58 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: awk "BEGIN {print \"/dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_b5d1434de12b984177d8fe3865f88829 /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_b5d1434de12b984177d8fe3865f88829 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:35:58 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: awk "BEGIN {print \"/dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_cde06dca54580b41329bf54d8306df4c /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_cde06dca54580b41329bf54d8306df4c xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:35:58 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_3f6e5bae8b9cfcddcc7d0f9b8ded99fc /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_3f6e5bae8b9cfcddcc7d0f9b8ded99fc
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:35:58 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_cde06dca54580b41329bf54d8306df4c /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_cde06dca54580b41329bf54d8306df4c
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:35:58 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_b5d1434de12b984177d8fe3865f88829 /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_b5d1434de12b984177d8fe3865f88829
>Result: 
>[negroni] Started GET /queue/138190be1330787bd37eed0a371f160f
>[negroni] Completed 200 OK in 144.072Âµs
>[kubeexec] DEBUG 2018/06/08 08:35:58 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_3f6e5bae8b9cfcddcc7d0f9b8ded99fc/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:35:58 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_cde06dca54580b41329bf54d8306df4c/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:35:58 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_b5d1434de12b984177d8fe3865f88829/brick
>Result: 
>[cmdexec] INFO 2018/06/08 08:35:58 Creating volume vol_b1f56e511cf62d1961997fb656989ed4 replica 3
>[kubeexec] DEBUG 2018/06/08 08:35:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script volume create vol_b1f56e511cf62d1961997fb656989ed4 replica 3 10.70.46.122:/var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_3f6e5bae8b9cfcddcc7d0f9b8ded99fc/brick 10.70.47.76:/var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_b5d1434de12b984177d8fe3865f88829/brick 10.70.46.187:/var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_cde06dca54580b41329bf54d8306df4c/brick
>Result: volume create: vol_b1f56e511cf62d1961997fb656989ed4: success: please start the volume to access data
>[negroni] Started GET /queue/138190be1330787bd37eed0a371f160f
>[negroni] Completed 200 OK in 147.726Âµs
>[negroni] Started GET /queue/138190be1330787bd37eed0a371f160f
>[negroni] Completed 200 OK in 158.24Âµs
>[negroni] Started GET /queue/138190be1330787bd37eed0a371f160f
>[negroni] Completed 200 OK in 133.548Âµs
>[negroni] Started GET /queue/138190be1330787bd37eed0a371f160f
>[negroni] Completed 200 OK in 317.474Âµs
>[kubeexec] DEBUG 2018/06/08 08:36:03 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script volume start vol_b1f56e511cf62d1961997fb656989ed4
>Result: volume start: vol_b1f56e511cf62d1961997fb656989ed4: success
>[heketi] INFO 2018/06/08 08:36:03 Create Volume succeeded
>[asynchttp] INFO 2018/06/08 08:36:03 asynchttp.go:292: Completed job 138190be1330787bd37eed0a371f160f in 5.593359534s
>[negroni] Started GET /queue/138190be1330787bd37eed0a371f160f
>[negroni] Completed 303 See Other in 170.532Âµs
>[negroni] Started GET /volumes/b1f56e511cf62d1961997fb656989ed4
>[negroni] Completed 200 OK in 2.948489ms
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 692.594Âµs
>[negroni] Started GET /volumes/27b4e68c05a8cded57c197df819b2c12
>[negroni] Completed 200 OK in 995.066Âµs
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 761.914Âµs
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 750.739Âµs
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 743.664Âµs
>[negroni] Started DELETE /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 409 Conflict in 539.611Âµs
>[negroni] Started GET /volumes/b1f56e511cf62d1961997fb656989ed4
>[negroni] Completed 200 OK in 1.111841ms
>[negroni] Started DELETE /volumes/27b4e68c05a8cded57c197df819b2c12
>[negroni] Completed 202 Accepted in 8.486765ms
>[asynchttp] INFO 2018/06/08 08:36:04 asynchttp.go:288: Started job 569c6cdffc1d7dc3a52c7f47162eb949
>[heketi] INFO 2018/06/08 08:36:04 Started async operation: Delete Volume
>[negroni] Started GET /queue/569c6cdffc1d7dc3a52c7f47162eb949
>[negroni] Completed 200 OK in 138.961Âµs
>[kubeexec] DEBUG 2018/06/08 08:36:04 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script snapshot list vol_27b4e68c05a8cded57c197df819b2c12 --xml
>Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
><cliOutput>
>  <opRet>0</opRet>
>  <opErrno>0</opErrno>
>  <opErrstr/>
>  <snapList>
>    <count>0</count>
>  </snapList>
></cliOutput>
>[negroni] Started GET /queue/569c6cdffc1d7dc3a52c7f47162eb949
>[negroni] Completed 200 OK in 201.286Âµs
>[negroni] Started GET /queue/569c6cdffc1d7dc3a52c7f47162eb949
>[negroni] Completed 200 OK in 164.305Âµs
>[kubeexec] DEBUG 2018/06/08 08:36:07 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script volume stop vol_27b4e68c05a8cded57c197df819b2c12 force
>Result: volume stop: vol_27b4e68c05a8cded57c197df819b2c12: success
>[kubeexec] DEBUG 2018/06/08 08:36:07 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script volume delete vol_27b4e68c05a8cded57c197df819b2c12
>Result: volume delete: vol_27b4e68c05a8cded57c197df819b2c12: success
>[heketi] INFO 2018/06/08 08:36:07 Deleting brick 16e82f0f7d6e040618ce84658b250ceb
>[heketi] INFO 2018/06/08 08:36:07 Deleting brick 89391ff92f83c755ffb9c270e0f6f3c9
>[heketi] INFO 2018/06/08 08:36:07 Deleting brick 40cd1137d297de071f2da2a433be6fd7
>[kubeexec] DEBUG 2018/06/08 08:36:07 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mount | grep -w /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_40cd1137d297de071f2da2a433be6fd7 | cut -d" " -f1
>Result: /dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_40cd1137d297de071f2da2a433be6fd7
>[kubeexec] DEBUG 2018/06/08 08:36:07 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mount | grep -w /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_16e82f0f7d6e040618ce84658b250ceb | cut -d" " -f1
>Result: /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_16e82f0f7d6e040618ce84658b250ceb
>[kubeexec] DEBUG 2018/06/08 08:36:07 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mount | grep -w /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_89391ff92f83c755ffb9c270e0f6f3c9 | cut -d" " -f1
>Result: /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_89391ff92f83c755ffb9c270e0f6f3c9
>[kubeexec] DEBUG 2018/06/08 08:36:07 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_40cd1137d297de071f2da2a433be6fd7
>
>Result:   vg_66a9af9f6bac95f9d8d556a2f14c29d3/tp_40cd1137d297de071f2da2a433be6fd7
>[kubeexec] DEBUG 2018/06/08 08:36:07 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_16e82f0f7d6e040618ce84658b250ceb
>
>Result:   vg_96f1667f2f1ced2c5ef94772922be93b/tp_16e82f0f7d6e040618ce84658b250ceb
>[negroni] Started GET /queue/569c6cdffc1d7dc3a52c7f47162eb949
>[negroni] Completed 200 OK in 132.66Âµs
>[kubeexec] DEBUG 2018/06/08 08:36:08 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_89391ff92f83c755ffb9c270e0f6f3c9
>
>Result:   vg_b861af9cc67f5bf8d5a8f9e38f56b14e/tp_89391ff92f83c755ffb9c270e0f6f3c9
>[negroni] Started GET /queue/569c6cdffc1d7dc3a52c7f47162eb949
>[negroni] Completed 200 OK in 142.032Âµs
>[kubeexec] DEBUG 2018/06/08 08:36:08 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: umount /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_40cd1137d297de071f2da2a433be6fd7
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:36:09 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: umount /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_16e82f0f7d6e040618ce84658b250ceb
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:36:09 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: umount /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_89391ff92f83c755ffb9c270e0f6f3c9
>Result: 
>[negroni] Started GET /queue/569c6cdffc1d7dc3a52c7f47162eb949
>[negroni] Completed 200 OK in 159.218Âµs
>[kubeexec] DEBUG 2018/06/08 08:36:10 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: sed -i.save "/brick_40cd1137d297de071f2da2a433be6fd7/d" /var/lib/heketi/fstab
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:36:10 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: sed -i.save "/brick_16e82f0f7d6e040618ce84658b250ceb/d" /var/lib/heketi/fstab
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:36:10 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: sed -i.save "/brick_89391ff92f83c755ffb9c270e0f6f3c9/d" /var/lib/heketi/fstab
>Result: 
>[negroni] Started GET /queue/569c6cdffc1d7dc3a52c7f47162eb949
>[negroni] Completed 200 OK in 197.817Âµs
>[kubeexec] DEBUG 2018/06/08 08:36:11 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvremove --autobackup=n -f /dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_40cd1137d297de071f2da2a433be6fd7
>
>Result:   Logical volume "brick_40cd1137d297de071f2da2a433be6fd7" successfully removed
>[kubeexec] DEBUG 2018/06/08 08:36:11 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvremove --autobackup=n -f /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_16e82f0f7d6e040618ce84658b250ceb
>
>Result:   Logical volume "brick_16e82f0f7d6e040618ce84658b250ceb" successfully removed
>[kubeexec] DEBUG 2018/06/08 08:36:11 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvremove --autobackup=n -f /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_89391ff92f83c755ffb9c270e0f6f3c9
>
>Result:   Logical volume "brick_89391ff92f83c755ffb9c270e0f6f3c9" successfully removed
>[negroni] Started GET /queue/569c6cdffc1d7dc3a52c7f47162eb949
>[negroni] Completed 200 OK in 312.448Âµs
>[kubeexec] DEBUG 2018/06/08 08:36:12 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvs --noheadings --options=thin_count   vg_66a9af9f6bac95f9d8d556a2f14c29d3/tp_40cd1137d297de071f2da2a433be6fd7
>
>Result:        0
>[kubeexec] DEBUG 2018/06/08 08:36:12 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvs --noheadings --options=thin_count   vg_96f1667f2f1ced2c5ef94772922be93b/tp_16e82f0f7d6e040618ce84658b250ceb
>
>Result:        0
>[kubeexec] DEBUG 2018/06/08 08:36:12 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvs --noheadings --options=thin_count   vg_b861af9cc67f5bf8d5a8f9e38f56b14e/tp_89391ff92f83c755ffb9c270e0f6f3c9
>
>Result:        0
>[negroni] Started GET /queue/569c6cdffc1d7dc3a52c7f47162eb949
>[negroni] Completed 200 OK in 158.543Âµs
>[kubeexec] DEBUG 2018/06/08 08:36:13 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvremove --autobackup=n -f   vg_66a9af9f6bac95f9d8d556a2f14c29d3/tp_40cd1137d297de071f2da2a433be6fd7
>
>Result:   Logical volume "tp_40cd1137d297de071f2da2a433be6fd7" successfully removed
>[negroni] Started GET /queue/569c6cdffc1d7dc3a52c7f47162eb949
>[negroni] Completed 200 OK in 140.218Âµs
>[kubeexec] DEBUG 2018/06/08 08:36:13 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvremove --autobackup=n -f   vg_96f1667f2f1ced2c5ef94772922be93b/tp_16e82f0f7d6e040618ce84658b250ceb
>
>Result:   Logical volume "tp_16e82f0f7d6e040618ce84658b250ceb" successfully removed
>[kubeexec] DEBUG 2018/06/08 08:36:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvremove --autobackup=n -f   vg_b861af9cc67f5bf8d5a8f9e38f56b14e/tp_89391ff92f83c755ffb9c270e0f6f3c9
>
>Result:   Logical volume "tp_89391ff92f83c755ffb9c270e0f6f3c9" successfully removed
>[kubeexec] DEBUG 2018/06/08 08:36:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: rmdir /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_40cd1137d297de071f2da2a433be6fd7
>Result: 
>[negroni] Started GET /queue/569c6cdffc1d7dc3a52c7f47162eb949
>[negroni] Completed 200 OK in 159.082Âµs
>[kubeexec] DEBUG 2018/06/08 08:36:15 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: rmdir /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_16e82f0f7d6e040618ce84658b250ceb
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:36:15 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: rmdir /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_89391ff92f83c755ffb9c270e0f6f3c9
>Result: 
>[heketi] INFO 2018/06/08 08:36:15 Delete Volume succeeded
>[asynchttp] INFO 2018/06/08 08:36:15 asynchttp.go:292: Completed job 569c6cdffc1d7dc3a52c7f47162eb949 in 10.434244873s
>[negroni] Started GET /queue/569c6cdffc1d7dc3a52c7f47162eb949
>[negroni] Completed 204 No Content in 161.738Âµs
>[negroni] Started DELETE /volumes/b1f56e511cf62d1961997fb656989ed4
>[negroni] Completed 202 Accepted in 10.799175ms
>[asynchttp] INFO 2018/06/08 08:36:15 asynchttp.go:288: Started job a2fe247c6d5feaa99ab8cc70b02d9dbd
>[heketi] INFO 2018/06/08 08:36:15 Started async operation: Delete Volume
>[negroni] Started GET /queue/a2fe247c6d5feaa99ab8cc70b02d9dbd
>[negroni] Completed 200 OK in 143.258Âµs
>[kubeexec] DEBUG 2018/06/08 08:36:16 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script snapshot list vol_b1f56e511cf62d1961997fb656989ed4 --xml
>Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
><cliOutput>
>  <opRet>0</opRet>
>  <opErrno>0</opErrno>
>  <opErrstr/>
>  <snapList>
>    <count>0</count>
>  </snapList>
></cliOutput>
>[negroni] Started GET /queue/a2fe247c6d5feaa99ab8cc70b02d9dbd
>[negroni] Completed 200 OK in 241.821Âµs
>[negroni] Started GET /queue/a2fe247c6d5feaa99ab8cc70b02d9dbd
>[negroni] Completed 200 OK in 305.617Âµs
>[kubeexec] DEBUG 2018/06/08 08:36:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script volume stop vol_b1f56e511cf62d1961997fb656989ed4 force
>Result: volume stop: vol_b1f56e511cf62d1961997fb656989ed4: success
>[kubeexec] DEBUG 2018/06/08 08:36:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script volume delete vol_b1f56e511cf62d1961997fb656989ed4
>Result: volume delete: vol_b1f56e511cf62d1961997fb656989ed4: success
>[heketi] INFO 2018/06/08 08:36:18 Deleting brick cde06dca54580b41329bf54d8306df4c
>[heketi] INFO 2018/06/08 08:36:18 Deleting brick b5d1434de12b984177d8fe3865f88829
>[heketi] INFO 2018/06/08 08:36:18 Deleting brick 3f6e5bae8b9cfcddcc7d0f9b8ded99fc
>[kubeexec] DEBUG 2018/06/08 08:36:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mount | grep -w /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_3f6e5bae8b9cfcddcc7d0f9b8ded99fc | cut -d" " -f1
>Result: /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_3f6e5bae8b9cfcddcc7d0f9b8ded99fc
>[kubeexec] DEBUG 2018/06/08 08:36:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mount | grep -w /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_cde06dca54580b41329bf54d8306df4c | cut -d" " -f1
>Result: /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_cde06dca54580b41329bf54d8306df4c
>[kubeexec] DEBUG 2018/06/08 08:36:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mount | grep -w /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_b5d1434de12b984177d8fe3865f88829 | cut -d" " -f1
>Result: /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_b5d1434de12b984177d8fe3865f88829
>[negroni] Started GET /queue/a2fe247c6d5feaa99ab8cc70b02d9dbd
>[negroni] Completed 200 OK in 168.658Âµs
>[kubeexec] DEBUG 2018/06/08 08:36:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_3f6e5bae8b9cfcddcc7d0f9b8ded99fc
>
>Result:   vg_3a4297677881963e3f80124971d50eea/tp_3f6e5bae8b9cfcddcc7d0f9b8ded99fc
>[kubeexec] DEBUG 2018/06/08 08:36:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_cde06dca54580b41329bf54d8306df4c
>
>Result:   vg_d389f0278a774bd7443a09af960961d8/tp_cde06dca54580b41329bf54d8306df4c
>[kubeexec] DEBUG 2018/06/08 08:36:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_b5d1434de12b984177d8fe3865f88829
>
>Result:   vg_9394bc70699b006c5460c9f654cf345f/tp_b5d1434de12b984177d8fe3865f88829
>[negroni] Started GET /queue/a2fe247c6d5feaa99ab8cc70b02d9dbd
>[negroni] Completed 200 OK in 148.03Âµs
>[kubeexec] DEBUG 2018/06/08 08:36:20 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: umount /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_3f6e5bae8b9cfcddcc7d0f9b8ded99fc
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:36:20 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: umount /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_cde06dca54580b41329bf54d8306df4c
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:36:20 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: umount /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_b5d1434de12b984177d8fe3865f88829
>Result: 
>[negroni] Started GET /queue/a2fe247c6d5feaa99ab8cc70b02d9dbd
>[negroni] Completed 200 OK in 232.815Âµs
>[kubeexec] DEBUG 2018/06/08 08:36:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: sed -i.save "/brick_3f6e5bae8b9cfcddcc7d0f9b8ded99fc/d" /var/lib/heketi/fstab
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:36:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: sed -i.save "/brick_cde06dca54580b41329bf54d8306df4c/d" /var/lib/heketi/fstab
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:36:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: sed -i.save "/brick_b5d1434de12b984177d8fe3865f88829/d" /var/lib/heketi/fstab
>Result: 
>[negroni] Started GET /queue/a2fe247c6d5feaa99ab8cc70b02d9dbd
>[negroni] Completed 200 OK in 178.399Âµs
>[kubeexec] DEBUG 2018/06/08 08:36:22 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvremove --autobackup=n -f /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_3f6e5bae8b9cfcddcc7d0f9b8ded99fc
>
>Result:   Logical volume "brick_3f6e5bae8b9cfcddcc7d0f9b8ded99fc" successfully removed
>[kubeexec] DEBUG 2018/06/08 08:36:22 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvremove --autobackup=n -f /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_cde06dca54580b41329bf54d8306df4c
>
>Result:   Logical volume "brick_cde06dca54580b41329bf54d8306df4c" successfully removed
>[kubeexec] DEBUG 2018/06/08 08:36:22 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvremove --autobackup=n -f /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_b5d1434de12b984177d8fe3865f88829
>
>Result:   Logical volume "brick_b5d1434de12b984177d8fe3865f88829" successfully removed
>[negroni] Started GET /queue/a2fe247c6d5feaa99ab8cc70b02d9dbd
>[negroni] Completed 200 OK in 163.057Âµs
>[kubeexec] DEBUG 2018/06/08 08:36:23 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvs --noheadings --options=thin_count   vg_3a4297677881963e3f80124971d50eea/tp_3f6e5bae8b9cfcddcc7d0f9b8ded99fc
>
>Result:        0
>[kubeexec] DEBUG 2018/06/08 08:36:23 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvs --noheadings --options=thin_count   vg_d389f0278a774bd7443a09af960961d8/tp_cde06dca54580b41329bf54d8306df4c
>
>Result:        0
>[negroni] Started GET /queue/a2fe247c6d5feaa99ab8cc70b02d9dbd
>[negroni] Completed 200 OK in 222.132Âµs
>[kubeexec] DEBUG 2018/06/08 08:36:24 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvs --noheadings --options=thin_count   vg_9394bc70699b006c5460c9f654cf345f/tp_b5d1434de12b984177d8fe3865f88829
>
>Result:        0
>[kubeexec] DEBUG 2018/06/08 08:36:24 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvremove --autobackup=n -f   vg_3a4297677881963e3f80124971d50eea/tp_3f6e5bae8b9cfcddcc7d0f9b8ded99fc
>
>Result:   Logical volume "tp_3f6e5bae8b9cfcddcc7d0f9b8ded99fc" successfully removed
>[negroni] Started GET /queue/a2fe247c6d5feaa99ab8cc70b02d9dbd
>[negroni] Completed 200 OK in 151.931Âµs
>[kubeexec] DEBUG 2018/06/08 08:36:25 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvremove --autobackup=n -f   vg_d389f0278a774bd7443a09af960961d8/tp_cde06dca54580b41329bf54d8306df4c
>
>Result:   Logical volume "tp_cde06dca54580b41329bf54d8306df4c" successfully removed
>[kubeexec] DEBUG 2018/06/08 08:36:25 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvremove --autobackup=n -f   vg_9394bc70699b006c5460c9f654cf345f/tp_b5d1434de12b984177d8fe3865f88829
>
>Result:   Logical volume "tp_b5d1434de12b984177d8fe3865f88829" successfully removed
>[negroni] Started GET /queue/a2fe247c6d5feaa99ab8cc70b02d9dbd
>[negroni] Completed 200 OK in 216.438Âµs
>[kubeexec] DEBUG 2018/06/08 08:36:26 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: rmdir /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_3f6e5bae8b9cfcddcc7d0f9b8ded99fc
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:36:26 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: rmdir /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_cde06dca54580b41329bf54d8306df4c
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:36:26 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: rmdir /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_b5d1434de12b984177d8fe3865f88829
>Result: 
>[heketi] INFO 2018/06/08 08:36:26 Delete Volume succeeded
>[asynchttp] INFO 2018/06/08 08:36:26 asynchttp.go:292: Completed job a2fe247c6d5feaa99ab8cc70b02d9dbd in 10.496159659s
>[negroni] Started GET /queue/a2fe247c6d5feaa99ab8cc70b02d9dbd
>[negroni] Completed 204 No Content in 150.149Âµs
>[negroni] Started GET /clusters
>[negroni] Completed 200 OK in 2.202189ms
>[negroni] Started GET /clusters/0a73c60efdd4673113b668afea101e6d
>[negroni] Completed 200 OK in 472.834Âµs
>[negroni] Started POST /volumes
>[heketi] INFO 2018/06/08 08:36:27 Allocating brick set #0
>[negroni] Completed 202 Accepted in 12.939698ms
>[asynchttp] INFO 2018/06/08 08:36:27 asynchttp.go:288: Started job 10cad4858b4d10d844dc008722f48815
>[heketi] INFO 2018/06/08 08:36:27 Started async operation: Create Volume
>[negroni] Started GET /queue/10cad4858b4d10d844dc008722f48815
>[negroni] Completed 200 OK in 169.307Âµs
>[heketi] INFO 2018/06/08 08:36:27 Creating brick 7895297323e5925bc0656d485ca1e227
>[heketi] INFO 2018/06/08 08:36:27 Creating brick 4c805bb73e44b37c40a402ffd4d57d64
>[heketi] INFO 2018/06/08 08:36:27 Creating brick 7163d990d4bfe7b8a4e7d72738f24672
>[kubeexec] DEBUG 2018/06/08 08:36:28 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir -p /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_4c805bb73e44b37c40a402ffd4d57d64
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:36:28 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir -p /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_7895297323e5925bc0656d485ca1e227
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:36:28 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir -p /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_7163d990d4bfe7b8a4e7d72738f24672
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:36:28 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvcreate --autobackup=n --poolmetadatasize 524288K --chunksize 256K --size 104857600K --thin vg_96f1667f2f1ced2c5ef94772922be93b/tp_7895297323e5925bc0656d485ca1e227 --virtualsize 104857600K --name brick_7895297323e5925bc0656d485ca1e227
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_7895297323e5925bc0656d485ca1e227" created.
>[kubeexec] DEBUG 2018/06/08 08:36:28 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvcreate --autobackup=n --poolmetadatasize 524288K --chunksize 256K --size 104857600K --thin vg_b861af9cc67f5bf8d5a8f9e38f56b14e/tp_7163d990d4bfe7b8a4e7d72738f24672 --virtualsize 104857600K --name brick_7163d990d4bfe7b8a4e7d72738f24672
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_7163d990d4bfe7b8a4e7d72738f24672" created.
>[kubeexec] DEBUG 2018/06/08 08:36:28 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvcreate --autobackup=n --poolmetadatasize 524288K --chunksize 256K --size 104857600K --thin vg_3a4297677881963e3f80124971d50eea/tp_4c805bb73e44b37c40a402ffd4d57d64 --virtualsize 104857600K --name brick_4c805bb73e44b37c40a402ffd4d57d64
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_4c805bb73e44b37c40a402ffd4d57d64" created.
>[kubeexec] DEBUG 2018/06/08 08:36:28 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_7163d990d4bfe7b8a4e7d72738f24672
>Result: meta-data=/dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_7163d990d4bfe7b8a4e7d72738f24672 isize=512    agcount=16, agsize=1638400 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=26214400, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=12800, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:36:28 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_4c805bb73e44b37c40a402ffd4d57d64
>Result: meta-data=/dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_4c805bb73e44b37c40a402ffd4d57d64 isize=512    agcount=16, agsize=1638400 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=26214400, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=12800, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[negroni] Started GET /queue/10cad4858b4d10d844dc008722f48815
>[negroni] Completed 200 OK in 131.16Âµs
>[kubeexec] DEBUG 2018/06/08 08:36:28 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: awk "BEGIN {print \"/dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_7163d990d4bfe7b8a4e7d72738f24672 /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_7163d990d4bfe7b8a4e7d72738f24672 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:36:29 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: awk "BEGIN {print \"/dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_4c805bb73e44b37c40a402ffd4d57d64 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_4c805bb73e44b37c40a402ffd4d57d64 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:36:29 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_7895297323e5925bc0656d485ca1e227
>Result: meta-data=/dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_7895297323e5925bc0656d485ca1e227 isize=512    agcount=16, agsize=1638400 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=26214400, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=12800, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:36:29 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_7163d990d4bfe7b8a4e7d72738f24672 /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_7163d990d4bfe7b8a4e7d72738f24672
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:36:29 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: awk "BEGIN {print \"/dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_7895297323e5925bc0656d485ca1e227 /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_7895297323e5925bc0656d485ca1e227 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:36:29 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_4c805bb73e44b37c40a402ffd4d57d64 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_4c805bb73e44b37c40a402ffd4d57d64
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:36:29 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_7163d990d4bfe7b8a4e7d72738f24672/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:36:29 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_4c805bb73e44b37c40a402ffd4d57d64/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:36:29 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_7895297323e5925bc0656d485ca1e227 /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_7895297323e5925bc0656d485ca1e227
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:36:29 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_7895297323e5925bc0656d485ca1e227/brick
>Result: 
>[cmdexec] INFO 2018/06/08 08:36:29 Creating volume vol_09be59d82458c2dc19aded51d4f93dae replica 3
>[kubeexec] DEBUG 2018/06/08 08:36:29 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script volume create vol_09be59d82458c2dc19aded51d4f93dae replica 3 10.70.47.76:/var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_7895297323e5925bc0656d485ca1e227/brick 10.70.46.187:/var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_7163d990d4bfe7b8a4e7d72738f24672/brick 10.70.46.122:/var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_4c805bb73e44b37c40a402ffd4d57d64/brick
>Result: volume create: vol_09be59d82458c2dc19aded51d4f93dae: success: please start the volume to access data
>[negroni] Started GET /queue/10cad4858b4d10d844dc008722f48815
>[negroni] Completed 200 OK in 135.074Âµs
>[negroni] Started GET /queue/10cad4858b4d10d844dc008722f48815
>[negroni] Completed 200 OK in 141.539Âµs
>[negroni] Started GET /queue/10cad4858b4d10d844dc008722f48815
>[negroni] Completed 200 OK in 142.953Âµs
>[kubeexec] DEBUG 2018/06/08 08:36:32 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script volume start vol_09be59d82458c2dc19aded51d4f93dae
>Result: volume start: vol_09be59d82458c2dc19aded51d4f93dae: success
>[heketi] INFO 2018/06/08 08:36:32 Create Volume succeeded
>[asynchttp] INFO 2018/06/08 08:36:32 asynchttp.go:292: Completed job 10cad4858b4d10d844dc008722f48815 in 4.969079534s
>[negroni] Started GET /queue/10cad4858b4d10d844dc008722f48815
>[negroni] Completed 303 See Other in 164.16Âµs
>[negroni] Started GET /volumes/09be59d82458c2dc19aded51d4f93dae
>[negroni] Completed 200 OK in 4.244428ms
>[negroni] Started POST /volumes
>[heketi] INFO 2018/06/08 08:36:33 Allocating brick set #0
>[heketi] INFO 2018/06/08 08:36:33 Allocating brick set #0
>[heketi] INFO 2018/06/08 08:36:33 Allocating brick set #1
>[heketi] INFO 2018/06/08 08:36:33 Allocating brick set #0
>[heketi] INFO 2018/06/08 08:36:33 Allocating brick set #1
>[heketi] INFO 2018/06/08 08:36:33 Allocating brick set #2
>[heketi] INFO 2018/06/08 08:36:33 Allocating brick set #0
>[heketi] INFO 2018/06/08 08:36:33 Allocating brick set #1
>[heketi] INFO 2018/06/08 08:36:33 Allocating brick set #2
>[heketi] INFO 2018/06/08 08:36:33 Allocating brick set #3
>[heketi] INFO 2018/06/08 08:36:33 Allocating brick set #4
>[heketi] INFO 2018/06/08 08:36:33 Allocating brick set #5
>[heketi] ERROR 2018/06/08 08:36:33 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:1167: Create Volume Build Failed: No space
>[negroni] Completed 500 Internal Server Error in 16.456422ms
>[negroni] Started GET /nodes/278bd6b4e16a8e62ef15aaae22e6abc1
>[negroni] Completed 200 OK in 3.139383ms
>[negroni] Started POST /devices
>[heketi] INFO 2018/06/08 08:36:33 Adding device /dev/sdf to node 278bd6b4e16a8e62ef15aaae22e6abc1
>[negroni] Completed 202 Accepted in 18.135157ms
>[asynchttp] INFO 2018/06/08 08:36:33 asynchttp.go:288: Started job a89144562a22950ecfca1a631b43e0e5
>[negroni] Started GET /queue/a89144562a22950ecfca1a631b43e0e5
>[negroni] Completed 200 OK in 115.114Âµs
>[kubeexec] DEBUG 2018/06/08 08:36:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: pvcreate --metadatasize=128M --dataalignment=256K '/dev/sdf'
>Result:   Physical volume "/dev/sdf" successfully created.
>[kubeexec] DEBUG 2018/06/08 08:36:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: vgcreate --autobackup=n vg_8d13a1486235f77a6360e21086e3d14b /dev/sdf
>Result:   Volume group "vg_8d13a1486235f77a6360e21086e3d14b" successfully created
>[kubeexec] DEBUG 2018/06/08 08:36:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: vgdisplay -c vg_8d13a1486235f77a6360e21086e3d14b
>Result:   vg_8d13a1486235f77a6360e21086e3d14b:r/w:772:-1:0:0:0:-1:0:1:1:104722432:4096:25567:0:25567:23PJ5D-S5P9-ZQEJ-KBzX-EQVe-fps2-6GxOCI
>[cmdexec] DEBUG 2018/06/08 08:36:33 /src/github.com/heketi/heketi/executors/cmdexec/device.go:147: Size of /dev/sdf in dhcp46-187.lab.eng.blr.redhat.com is 104722432
>[heketi] INFO 2018/06/08 08:36:33 Added device /dev/sdf
>[asynchttp] INFO 2018/06/08 08:36:33 asynchttp.go:292: Completed job a89144562a22950ecfca1a631b43e0e5 in 551.429126ms
>[negroni] Started GET /queue/a89144562a22950ecfca1a631b43e0e5
>[negroni] Completed 204 No Content in 153.126Âµs
>[negroni] Started GET /nodes/278bd6b4e16a8e62ef15aaae22e6abc1
>[negroni] Completed 200 OK in 5.417209ms
>[negroni] Started GET /nodes/70423cc0bcd044fe5ba8bbbd256a3e49
>[negroni] Completed 200 OK in 816.98Âµs
>[negroni] Started POST /devices
>[heketi] INFO 2018/06/08 08:36:34 Adding device /dev/sdf to node 70423cc0bcd044fe5ba8bbbd256a3e49
>[negroni] Completed 202 Accepted in 18.845189ms
>[asynchttp] INFO 2018/06/08 08:36:34 asynchttp.go:288: Started job f0326b30b43032954e772250785ae5c6
>[negroni] Started GET /queue/f0326b30b43032954e772250785ae5c6
>[negroni] Completed 200 OK in 108.713Âµs
>[kubeexec] DEBUG 2018/06/08 08:36:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: pvcreate --metadatasize=128M --dataalignment=256K '/dev/sdf'
>Result:   Physical volume "/dev/sdf" successfully created.
>[kubeexec] DEBUG 2018/06/08 08:36:35 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: vgcreate --autobackup=n vg_e883b6f7d90329384bba2aeb2c9a34b6 /dev/sdf
>Result:   Volume group "vg_e883b6f7d90329384bba2aeb2c9a34b6" successfully created
>[kubeexec] DEBUG 2018/06/08 08:36:35 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: vgdisplay -c vg_e883b6f7d90329384bba2aeb2c9a34b6
>Result:   vg_e883b6f7d90329384bba2aeb2c9a34b6:r/w:772:-1:0:0:0:-1:0:1:1:104722432:4096:25567:0:25567:szmF91-22BX-RzJM-zAiK-Aqsk-Idy5-Du4dpC
>[cmdexec] DEBUG 2018/06/08 08:36:35 /src/github.com/heketi/heketi/executors/cmdexec/device.go:147: Size of /dev/sdf in dhcp46-122.lab.eng.blr.redhat.com is 104722432
>[heketi] INFO 2018/06/08 08:36:35 Added device /dev/sdf
>[asynchttp] INFO 2018/06/08 08:36:35 asynchttp.go:292: Completed job f0326b30b43032954e772250785ae5c6 in 462.499304ms
>[negroni] Started GET /queue/f0326b30b43032954e772250785ae5c6
>[negroni] Completed 204 No Content in 151.289Âµs
>[negroni] Started GET /nodes/70423cc0bcd044fe5ba8bbbd256a3e49
>[negroni] Completed 200 OK in 7.455775ms
>[negroni] Started GET /nodes/d942fe6c0ee5691b7cc263968f97b650
>[negroni] Completed 200 OK in 928.367Âµs
>[negroni] Started POST /devices
>[heketi] INFO 2018/06/08 08:36:36 Adding device /dev/sdf to node d942fe6c0ee5691b7cc263968f97b650
>[negroni] Completed 202 Accepted in 12.903779ms
>[asynchttp] INFO 2018/06/08 08:36:36 asynchttp.go:288: Started job e7deadfa5ef6fdfee306d153c63045c6
>[negroni] Started GET /queue/e7deadfa5ef6fdfee306d153c63045c6
>[negroni] Completed 200 OK in 177.652Âµs
>[kubeexec] DEBUG 2018/06/08 08:36:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: pvcreate --metadatasize=128M --dataalignment=256K '/dev/sdf'
>Result:   Physical volume "/dev/sdf" successfully created.
>[kubeexec] DEBUG 2018/06/08 08:36:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: vgcreate --autobackup=n vg_abadb4ba1e7781b57e43a25ef34a08cc /dev/sdf
>Result:   Volume group "vg_abadb4ba1e7781b57e43a25ef34a08cc" successfully created
>[kubeexec] DEBUG 2018/06/08 08:36:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: vgdisplay -c vg_abadb4ba1e7781b57e43a25ef34a08cc
>Result:   vg_abadb4ba1e7781b57e43a25ef34a08cc:r/w:772:-1:0:0:0:-1:0:1:1:104722432:4096:25567:0:25567:MS7mS3-HAHh-ufnf-2cmn-lZAt-TrCi-pHR9W2
>[cmdexec] DEBUG 2018/06/08 08:36:36 /src/github.com/heketi/heketi/executors/cmdexec/device.go:147: Size of /dev/sdf in dhcp47-76.lab.eng.blr.redhat.com is 104722432
>[heketi] INFO 2018/06/08 08:36:36 Added device /dev/sdf
>[asynchttp] INFO 2018/06/08 08:36:36 asynchttp.go:292: Completed job e7deadfa5ef6fdfee306d153c63045c6 in 556.314563ms
>[negroni] Started GET /queue/e7deadfa5ef6fdfee306d153c63045c6
>[negroni] Completed 204 No Content in 128.048Âµs
>[negroni] Started GET /nodes/d942fe6c0ee5691b7cc263968f97b650
>[negroni] Completed 200 OK in 5.778053ms
>[negroni] Started POST /volumes
>[heketi] INFO 2018/06/08 08:36:37 Allocating brick set #0
>[heketi] INFO 2018/06/08 08:36:37 Allocating brick set #0
>[heketi] INFO 2018/06/08 08:36:37 Allocating brick set #1
>[heketi] INFO 2018/06/08 08:36:37 Allocating brick set #0
>[heketi] INFO 2018/06/08 08:36:37 Allocating brick set #1
>[heketi] INFO 2018/06/08 08:36:37 Allocating brick set #2
>[heketi] INFO 2018/06/08 08:36:37 Allocating brick set #0
>[heketi] INFO 2018/06/08 08:36:37 Allocating brick set #1
>[heketi] INFO 2018/06/08 08:36:37 Allocating brick set #2
>[heketi] INFO 2018/06/08 08:36:37 Allocating brick set #3
>[heketi] INFO 2018/06/08 08:36:37 Allocating brick set #4
>[heketi] INFO 2018/06/08 08:36:37 Allocating brick set #5
>[heketi] INFO 2018/06/08 08:36:37 Allocating brick set #6
>[negroni] Completed 500 Internal Server Error in 19.447586ms
>[heketi] ERROR 2018/06/08 08:36:37 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:1167: Create Volume Build Failed: No space
>[negroni] Started POST /devices/8d13a1486235f77a6360e21086e3d14b/state
>[negroni] Completed 202 Accepted in 441.559Âµs
>[asynchttp] INFO 2018/06/08 08:36:37 asynchttp.go:288: Started job 27db2f5d4acee89343326162fa652ac3
>[negroni] Started GET /queue/27db2f5d4acee89343326162fa652ac3
>[negroni] Completed 200 OK in 135.744Âµs
>[asynchttp] INFO 2018/06/08 08:36:37 asynchttp.go:292: Completed job 27db2f5d4acee89343326162fa652ac3 in 7.904311ms
>[negroni] Started GET /queue/27db2f5d4acee89343326162fa652ac3
>[negroni] Completed 204 No Content in 139.829Âµs
>[negroni] Started POST /devices/8d13a1486235f77a6360e21086e3d14b/state
>[negroni] Completed 202 Accepted in 3.726191ms
>[asynchttp] INFO 2018/06/08 08:36:38 asynchttp.go:288: Started job 7013c42e401275e0fd4a27333367afe1
>[heketi] INFO 2018/06/08 08:36:38 Running Remove Device
>[negroni] Started GET /queue/7013c42e401275e0fd4a27333367afe1
>[negroni] Completed 200 OK in 172.918Âµs
>[asynchttp] INFO 2018/06/08 08:36:38 asynchttp.go:292: Completed job 7013c42e401275e0fd4a27333367afe1 in 13.175734ms
>[negroni] Started GET /queue/7013c42e401275e0fd4a27333367afe1
>[negroni] Completed 204 No Content in 219.178Âµs
>[negroni] Started DELETE /devices/8d13a1486235f77a6360e21086e3d14b
>[heketi] INFO 2018/06/08 08:36:39 Deleting device 8d13a1486235f77a6360e21086e3d14b on node 278bd6b4e16a8e62ef15aaae22e6abc1
>[negroni] Completed 202 Accepted in 1.531976ms
>[asynchttp] INFO 2018/06/08 08:36:39 asynchttp.go:288: Started job 70b67b8f1fe31c7045ad0415fd4e36e9
>[negroni] Started GET /queue/70b67b8f1fe31c7045ad0415fd4e36e9
>[negroni] Completed 200 OK in 180.419Âµs
>[kubeexec] DEBUG 2018/06/08 08:36:39 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: vgremove vg_8d13a1486235f77a6360e21086e3d14b
>Result:   Volume group "vg_8d13a1486235f77a6360e21086e3d14b" successfully removed
>[kubeexec] DEBUG 2018/06/08 08:36:39 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: pvremove '/dev/sdf'
>Result:   Labels on physical volume "/dev/sdf" successfully wiped.
>[kubeexec] ERROR 2018/06/08 08:36:40 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [ls /var/lib/heketi/mounts/vg_8d13a1486235f77a6360e21086e3d14b] on glusterfs-storage-vsh2m: Err[command terminated with exit code 2]: Stdout []: Stderr [ls: cannot access /var/lib/heketi/mounts/vg_8d13a1486235f77a6360e21086e3d14b: No such file or directory
>]
>[heketi] INFO 2018/06/08 08:36:40 Deleted node [8d13a1486235f77a6360e21086e3d14b]
>[asynchttp] INFO 2018/06/08 08:36:40 asynchttp.go:292: Completed job 70b67b8f1fe31c7045ad0415fd4e36e9 in 435.186345ms
>[negroni] Started GET /queue/70b67b8f1fe31c7045ad0415fd4e36e9
>[negroni] Completed 204 No Content in 216.916Âµs
>[negroni] Started POST /devices/e883b6f7d90329384bba2aeb2c9a34b6/state
>[negroni] Completed 202 Accepted in 4.157877ms
>[asynchttp] INFO 2018/06/08 08:36:40 asynchttp.go:288: Started job 3115668335a67e168219486afcf5b70d
>[negroni] Started GET /queue/3115668335a67e168219486afcf5b70d
>[negroni] Completed 200 OK in 194.522Âµs
>[asynchttp] INFO 2018/06/08 08:36:40 asynchttp.go:292: Completed job 3115668335a67e168219486afcf5b70d in 9.442688ms
>[negroni] Started GET /queue/3115668335a67e168219486afcf5b70d
>[negroni] Completed 204 No Content in 190.105Âµs
>[negroni] Started POST /devices/e883b6f7d90329384bba2aeb2c9a34b6/state
>[negroni] Completed 202 Accepted in 3.353577ms
>[asynchttp] INFO 2018/06/08 08:36:41 asynchttp.go:288: Started job f23d9e4ef280417862547668aea83868
>[heketi] INFO 2018/06/08 08:36:41 Running Remove Device
>[negroni] Started GET /queue/f23d9e4ef280417862547668aea83868
>[negroni] Completed 200 OK in 163.723Âµs
>[asynchttp] INFO 2018/06/08 08:36:41 asynchttp.go:292: Completed job f23d9e4ef280417862547668aea83868 in 22.396784ms
>[negroni] Started GET /queue/f23d9e4ef280417862547668aea83868
>[negroni] Completed 204 No Content in 185.333Âµs
>[negroni] Started DELETE /devices/e883b6f7d90329384bba2aeb2c9a34b6
>[heketi] INFO 2018/06/08 08:36:43 Deleting device e883b6f7d90329384bba2aeb2c9a34b6 on node 70423cc0bcd044fe5ba8bbbd256a3e49
>[negroni] Completed 202 Accepted in 1.496931ms
>[asynchttp] INFO 2018/06/08 08:36:43 asynchttp.go:288: Started job d3076912264b8a47eda6607a096b1636
>[negroni] Started GET /queue/d3076912264b8a47eda6607a096b1636
>[negroni] Completed 200 OK in 127.117Âµs
>[kubeexec] DEBUG 2018/06/08 08:36:43 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: vgremove vg_e883b6f7d90329384bba2aeb2c9a34b6
>Result:   Volume group "vg_e883b6f7d90329384bba2aeb2c9a34b6" successfully removed
>[kubeexec] DEBUG 2018/06/08 08:36:43 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: pvremove '/dev/sdf'
>Result:   Labels on physical volume "/dev/sdf" successfully wiped.
>[kubeexec] ERROR 2018/06/08 08:36:43 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [ls /var/lib/heketi/mounts/vg_e883b6f7d90329384bba2aeb2c9a34b6] on glusterfs-storage-pg4xc: Err[command terminated with exit code 2]: Stdout []: Stderr [ls: cannot access /var/lib/heketi/mounts/vg_e883b6f7d90329384bba2aeb2c9a34b6: No such file or directory
>]
>[heketi] INFO 2018/06/08 08:36:43 Deleted node [e883b6f7d90329384bba2aeb2c9a34b6]
>[asynchttp] INFO 2018/06/08 08:36:43 asynchttp.go:292: Completed job d3076912264b8a47eda6607a096b1636 in 459.923605ms
>[negroni] Started GET /queue/d3076912264b8a47eda6607a096b1636
>[negroni] Completed 204 No Content in 138.955Âµs
>[negroni] Started POST /devices/abadb4ba1e7781b57e43a25ef34a08cc/state
>[negroni] Completed 202 Accepted in 3.762531ms
>[asynchttp] INFO 2018/06/08 08:36:44 asynchttp.go:288: Started job 3ac746dece871332cf300128197e54d6
>[negroni] Started GET /queue/3ac746dece871332cf300128197e54d6
>[negroni] Completed 200 OK in 180.473Âµs
>[asynchttp] INFO 2018/06/08 08:36:44 asynchttp.go:292: Completed job 3ac746dece871332cf300128197e54d6 in 10.8561ms
>[negroni] Started GET /queue/3ac746dece871332cf300128197e54d6
>[negroni] Completed 204 No Content in 160.731Âµs
>[negroni] Started POST /devices/abadb4ba1e7781b57e43a25ef34a08cc/state
>[negroni] Completed 202 Accepted in 3.89285ms
>[asynchttp] INFO 2018/06/08 08:36:45 asynchttp.go:288: Started job c63b6a3cd977592e26f919e8c94751a4
>[heketi] INFO 2018/06/08 08:36:45 Running Remove Device
>[negroni] Started GET /queue/c63b6a3cd977592e26f919e8c94751a4
>[negroni] Completed 200 OK in 160.225Âµs
>[asynchttp] INFO 2018/06/08 08:36:45 asynchttp.go:292: Completed job c63b6a3cd977592e26f919e8c94751a4 in 14.303126ms
>[negroni] Started GET /queue/c63b6a3cd977592e26f919e8c94751a4
>[negroni] Completed 204 No Content in 182.061Âµs
>[negroni] Started DELETE /devices/abadb4ba1e7781b57e43a25ef34a08cc
>[heketi] INFO 2018/06/08 08:36:46 Deleting device abadb4ba1e7781b57e43a25ef34a08cc on node d942fe6c0ee5691b7cc263968f97b650
>[negroni] Completed 202 Accepted in 1.318764ms
>[asynchttp] INFO 2018/06/08 08:36:46 asynchttp.go:288: Started job ce6706f0f75d5d4bd8d83414d79d93db
>[negroni] Started GET /queue/ce6706f0f75d5d4bd8d83414d79d93db
>[negroni] Completed 200 OK in 298.93Âµs
>[kubeexec] DEBUG 2018/06/08 08:36:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: vgremove vg_abadb4ba1e7781b57e43a25ef34a08cc
>Result:   Volume group "vg_abadb4ba1e7781b57e43a25ef34a08cc" successfully removed
>[kubeexec] DEBUG 2018/06/08 08:36:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: pvremove '/dev/sdf'
>Result:   Labels on physical volume "/dev/sdf" successfully wiped.
>[kubeexec] ERROR 2018/06/08 08:36:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [ls /var/lib/heketi/mounts/vg_abadb4ba1e7781b57e43a25ef34a08cc] on glusterfs-storage-gxp7c: Err[command terminated with exit code 2]: Stdout []: Stderr [ls: cannot access /var/lib/heketi/mounts/vg_abadb4ba1e7781b57e43a25ef34a08cc: No such file or directory
>]
>[heketi] INFO 2018/06/08 08:36:46 Deleted node [abadb4ba1e7781b57e43a25ef34a08cc]
>[asynchttp] INFO 2018/06/08 08:36:46 asynchttp.go:292: Completed job ce6706f0f75d5d4bd8d83414d79d93db in 480.222279ms
>[negroni] Started GET /queue/ce6706f0f75d5d4bd8d83414d79d93db
>[negroni] Completed 204 No Content in 201.273Âµs
>[negroni] Started DELETE /volumes/09be59d82458c2dc19aded51d4f93dae
>[negroni] Completed 202 Accepted in 13.760757ms
>[asynchttp] INFO 2018/06/08 08:36:47 asynchttp.go:288: Started job 7a23b9eb593cf5579b7ed54e9f496268
>[heketi] INFO 2018/06/08 08:36:47 Started async operation: Delete Volume
>[negroni] Started GET /queue/7a23b9eb593cf5579b7ed54e9f496268
>[negroni] Completed 200 OK in 185.475Âµs
>[kubeexec] DEBUG 2018/06/08 08:36:47 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script snapshot list vol_09be59d82458c2dc19aded51d4f93dae --xml
>Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
><cliOutput>
>  <opRet>0</opRet>
>  <opErrno>0</opErrno>
>  <opErrstr/>
>  <snapList>
>    <count>0</count>
>  </snapList>
></cliOutput>
>[negroni] Started GET /queue/7a23b9eb593cf5579b7ed54e9f496268
>[negroni] Completed 200 OK in 182.002Âµs
>[negroni] Started GET /queue/7a23b9eb593cf5579b7ed54e9f496268
>[negroni] Completed 200 OK in 181.004Âµs
>[kubeexec] DEBUG 2018/06/08 08:36:50 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script volume stop vol_09be59d82458c2dc19aded51d4f93dae force
>Result: volume stop: vol_09be59d82458c2dc19aded51d4f93dae: success
>[kubeexec] DEBUG 2018/06/08 08:36:50 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script volume delete vol_09be59d82458c2dc19aded51d4f93dae
>Result: volume delete: vol_09be59d82458c2dc19aded51d4f93dae: success
>[heketi] INFO 2018/06/08 08:36:50 Deleting brick 4c805bb73e44b37c40a402ffd4d57d64
>[heketi] INFO 2018/06/08 08:36:50 Deleting brick 7163d990d4bfe7b8a4e7d72738f24672
>[heketi] INFO 2018/06/08 08:36:50 Deleting brick 7895297323e5925bc0656d485ca1e227
>[kubeexec] DEBUG 2018/06/08 08:36:50 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mount | grep -w /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_4c805bb73e44b37c40a402ffd4d57d64 | cut -d" " -f1
>Result: /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_4c805bb73e44b37c40a402ffd4d57d64
>[kubeexec] DEBUG 2018/06/08 08:36:50 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mount | grep -w /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_7163d990d4bfe7b8a4e7d72738f24672 | cut -d" " -f1
>Result: /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_7163d990d4bfe7b8a4e7d72738f24672
>[kubeexec] DEBUG 2018/06/08 08:36:50 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mount | grep -w /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_7895297323e5925bc0656d485ca1e227 | cut -d" " -f1
>Result: /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_7895297323e5925bc0656d485ca1e227
>[negroni] Started GET /queue/7a23b9eb593cf5579b7ed54e9f496268
>[negroni] Completed 200 OK in 312.857Âµs
>[kubeexec] DEBUG 2018/06/08 08:36:50 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_7163d990d4bfe7b8a4e7d72738f24672
>
>Result:   vg_b861af9cc67f5bf8d5a8f9e38f56b14e/tp_7163d990d4bfe7b8a4e7d72738f24672
>[kubeexec] DEBUG 2018/06/08 08:36:50 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_4c805bb73e44b37c40a402ffd4d57d64
>
>Result:   vg_3a4297677881963e3f80124971d50eea/tp_4c805bb73e44b37c40a402ffd4d57d64
>[kubeexec] DEBUG 2018/06/08 08:36:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_7895297323e5925bc0656d485ca1e227
>
>Result:   vg_96f1667f2f1ced2c5ef94772922be93b/tp_7895297323e5925bc0656d485ca1e227
>[negroni] Started GET /queue/7a23b9eb593cf5579b7ed54e9f496268
>[negroni] Completed 200 OK in 245.272Âµs
>[kubeexec] DEBUG 2018/06/08 08:36:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: umount /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_7163d990d4bfe7b8a4e7d72738f24672
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:36:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: umount /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_4c805bb73e44b37c40a402ffd4d57d64
>Result: 
>[negroni] Started GET /queue/7a23b9eb593cf5579b7ed54e9f496268
>[negroni] Completed 200 OK in 128.077Âµs
>[kubeexec] DEBUG 2018/06/08 08:36:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: umount /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_7895297323e5925bc0656d485ca1e227
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:36:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: sed -i.save "/brick_7163d990d4bfe7b8a4e7d72738f24672/d" /var/lib/heketi/fstab
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:36:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: sed -i.save "/brick_4c805bb73e44b37c40a402ffd4d57d64/d" /var/lib/heketi/fstab
>Result: 
>[negroni] Started GET /queue/7a23b9eb593cf5579b7ed54e9f496268
>[negroni] Completed 200 OK in 196.706Âµs
>[kubeexec] DEBUG 2018/06/08 08:36:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: sed -i.save "/brick_7895297323e5925bc0656d485ca1e227/d" /var/lib/heketi/fstab
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:36:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvremove --autobackup=n -f /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_7163d990d4bfe7b8a4e7d72738f24672
>
>Result:   Logical volume "brick_7163d990d4bfe7b8a4e7d72738f24672" successfully removed
>[kubeexec] DEBUG 2018/06/08 08:36:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvremove --autobackup=n -f /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_4c805bb73e44b37c40a402ffd4d57d64
>
>Result:   Logical volume "brick_4c805bb73e44b37c40a402ffd4d57d64" successfully removed
>[negroni] Started GET /queue/7a23b9eb593cf5579b7ed54e9f496268
>[negroni] Completed 200 OK in 160.827Âµs
>[kubeexec] DEBUG 2018/06/08 08:36:55 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvremove --autobackup=n -f /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_7895297323e5925bc0656d485ca1e227
>
>Result:   Logical volume "brick_7895297323e5925bc0656d485ca1e227" successfully removed
>[kubeexec] DEBUG 2018/06/08 08:36:55 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvs --noheadings --options=thin_count   vg_b861af9cc67f5bf8d5a8f9e38f56b14e/tp_7163d990d4bfe7b8a4e7d72738f24672
>
>Result:        0
>[kubeexec] DEBUG 2018/06/08 08:36:55 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvs --noheadings --options=thin_count   vg_3a4297677881963e3f80124971d50eea/tp_4c805bb73e44b37c40a402ffd4d57d64
>
>Result:        0
>[negroni] Started GET /queue/7a23b9eb593cf5579b7ed54e9f496268
>[negroni] Completed 200 OK in 197.665Âµs
>[kubeexec] DEBUG 2018/06/08 08:36:56 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvs --noheadings --options=thin_count   vg_96f1667f2f1ced2c5ef94772922be93b/tp_7895297323e5925bc0656d485ca1e227
>
>Result:        0
>[kubeexec] DEBUG 2018/06/08 08:36:56 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvremove --autobackup=n -f   vg_b861af9cc67f5bf8d5a8f9e38f56b14e/tp_7163d990d4bfe7b8a4e7d72738f24672
>
>Result:   Logical volume "tp_7163d990d4bfe7b8a4e7d72738f24672" successfully removed
>[negroni] Started GET /queue/7a23b9eb593cf5579b7ed54e9f496268
>[negroni] Completed 200 OK in 229.258Âµs
>[kubeexec] DEBUG 2018/06/08 08:36:56 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvremove --autobackup=n -f   vg_3a4297677881963e3f80124971d50eea/tp_4c805bb73e44b37c40a402ffd4d57d64
>
>Result:   Logical volume "tp_4c805bb73e44b37c40a402ffd4d57d64" successfully removed
>[kubeexec] DEBUG 2018/06/08 08:36:57 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvremove --autobackup=n -f   vg_96f1667f2f1ced2c5ef94772922be93b/tp_7895297323e5925bc0656d485ca1e227
>
>Result:   Logical volume "tp_7895297323e5925bc0656d485ca1e227" successfully removed
>[negroni] Started GET /queue/7a23b9eb593cf5579b7ed54e9f496268
>[negroni] Completed 200 OK in 215.885Âµs
>[kubeexec] DEBUG 2018/06/08 08:36:57 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: rmdir /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_7163d990d4bfe7b8a4e7d72738f24672
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:36:57 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: rmdir /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_4c805bb73e44b37c40a402ffd4d57d64
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:36:58 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: rmdir /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_7895297323e5925bc0656d485ca1e227
>Result: 
>[heketi] INFO 2018/06/08 08:36:58 Delete Volume succeeded
>[asynchttp] INFO 2018/06/08 08:36:58 asynchttp.go:292: Completed job 7a23b9eb593cf5579b7ed54e9f496268 in 10.667741714s
>[negroni] Started GET /queue/7a23b9eb593cf5579b7ed54e9f496268
>[negroni] Completed 204 No Content in 226.127Âµs
>[negroni] Started POST /volumes
>[heketi] INFO 2018/06/08 08:36:58 Allocating brick set #0
>[negroni] Completed 202 Accepted in 17.409592ms
>[asynchttp] INFO 2018/06/08 08:36:58 asynchttp.go:288: Started job 6cfc77e324d4cbd460708e0cad397c9c
>[heketi] INFO 2018/06/08 08:36:58 Started async operation: Create Volume
>[negroni] Started GET /queue/6cfc77e324d4cbd460708e0cad397c9c
>[negroni] Completed 200 OK in 124.474Âµs
>[heketi] INFO 2018/06/08 08:36:58 Creating brick f526234df2e4ac0fa1c80096c3759590
>[heketi] INFO 2018/06/08 08:36:58 Creating brick 6ba663d12286dbed5c5caa2b3d947f69
>[heketi] INFO 2018/06/08 08:36:58 Creating brick 07da7af9f0e00909886f0c79a0eef485
>[kubeexec] DEBUG 2018/06/08 08:36:58 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir -p /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_f526234df2e4ac0fa1c80096c3759590
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:36:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir -p /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_07da7af9f0e00909886f0c79a0eef485
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:36:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvcreate --autobackup=n --poolmetadatasize 53248K --chunksize 256K --size 10485760K --thin vg_3a4297677881963e3f80124971d50eea/tp_f526234df2e4ac0fa1c80096c3759590 --virtualsize 10485760K --name brick_f526234df2e4ac0fa1c80096c3759590
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_f526234df2e4ac0fa1c80096c3759590" created.
>[kubeexec] DEBUG 2018/06/08 08:36:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir -p /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_6ba663d12286dbed5c5caa2b3d947f69
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:36:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_f526234df2e4ac0fa1c80096c3759590
>Result: meta-data=/dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_f526234df2e4ac0fa1c80096c3759590 isize=512    agcount=16, agsize=163840 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=2621440, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:36:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvcreate --autobackup=n --poolmetadatasize 53248K --chunksize 256K --size 10485760K --thin vg_d389f0278a774bd7443a09af960961d8/tp_07da7af9f0e00909886f0c79a0eef485 --virtualsize 10485760K --name brick_07da7af9f0e00909886f0c79a0eef485
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_07da7af9f0e00909886f0c79a0eef485" created.
>[kubeexec] DEBUG 2018/06/08 08:36:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: awk "BEGIN {print \"/dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_f526234df2e4ac0fa1c80096c3759590 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_f526234df2e4ac0fa1c80096c3759590 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:36:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_07da7af9f0e00909886f0c79a0eef485
>Result: meta-data=/dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_07da7af9f0e00909886f0c79a0eef485 isize=512    agcount=16, agsize=163840 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=2621440, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:36:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_f526234df2e4ac0fa1c80096c3759590 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_f526234df2e4ac0fa1c80096c3759590
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:36:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvcreate --autobackup=n --poolmetadatasize 53248K --chunksize 256K --size 10485760K --thin vg_9394bc70699b006c5460c9f654cf345f/tp_6ba663d12286dbed5c5caa2b3d947f69 --virtualsize 10485760K --name brick_6ba663d12286dbed5c5caa2b3d947f69
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_6ba663d12286dbed5c5caa2b3d947f69" created.
>[negroni] Started GET /queue/6cfc77e324d4cbd460708e0cad397c9c
>[negroni] Completed 200 OK in 118.69Âµs
>[kubeexec] DEBUG 2018/06/08 08:36:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: awk "BEGIN {print \"/dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_07da7af9f0e00909886f0c79a0eef485 /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_07da7af9f0e00909886f0c79a0eef485 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:36:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_f526234df2e4ac0fa1c80096c3759590/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:36:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_6ba663d12286dbed5c5caa2b3d947f69
>Result: meta-data=/dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_6ba663d12286dbed5c5caa2b3d947f69 isize=512    agcount=16, agsize=163840 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=2621440, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:36:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_07da7af9f0e00909886f0c79a0eef485 /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_07da7af9f0e00909886f0c79a0eef485
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:37:00 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: awk "BEGIN {print \"/dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_6ba663d12286dbed5c5caa2b3d947f69 /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_6ba663d12286dbed5c5caa2b3d947f69 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:37:00 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_07da7af9f0e00909886f0c79a0eef485/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:37:00 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_6ba663d12286dbed5c5caa2b3d947f69 /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_6ba663d12286dbed5c5caa2b3d947f69
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:37:00 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_6ba663d12286dbed5c5caa2b3d947f69/brick
>Result: 
>[cmdexec] INFO 2018/06/08 08:37:00 Creating volume vol_a29f96e611e83fe05af97f0b595cb460 replica 3
>[kubeexec] DEBUG 2018/06/08 08:37:00 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script volume create vol_a29f96e611e83fe05af97f0b595cb460 replica 3 10.70.46.122:/var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_f526234df2e4ac0fa1c80096c3759590/brick 10.70.47.76:/var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_6ba663d12286dbed5c5caa2b3d947f69/brick 10.70.46.187:/var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_07da7af9f0e00909886f0c79a0eef485/brick
>Result: volume create: vol_a29f96e611e83fe05af97f0b595cb460: success: please start the volume to access data
>[negroni] Started GET /queue/6cfc77e324d4cbd460708e0cad397c9c
>[negroni] Completed 200 OK in 151.983Âµs
>[negroni] Started GET /queue/6cfc77e324d4cbd460708e0cad397c9c
>[negroni] Completed 200 OK in 131.357Âµs
>[kubeexec] DEBUG 2018/06/08 08:37:02 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script volume start vol_a29f96e611e83fe05af97f0b595cb460
>Result: volume start: vol_a29f96e611e83fe05af97f0b595cb460: success
>[heketi] INFO 2018/06/08 08:37:02 Create Volume succeeded
>[asynchttp] INFO 2018/06/08 08:37:02 asynchttp.go:292: Completed job 6cfc77e324d4cbd460708e0cad397c9c in 3.990543659s
>[negroni] Started GET /queue/6cfc77e324d4cbd460708e0cad397c9c
>[negroni] Completed 303 See Other in 199.362Âµs
>[negroni] Started GET /volumes/a29f96e611e83fe05af97f0b595cb460
>[negroni] Completed 200 OK in 5.749052ms
>[negroni] Started GET /clusters
>[negroni] Completed 200 OK in 171.625Âµs
>[negroni] Started GET /clusters/0a73c60efdd4673113b668afea101e6d
>[negroni] Completed 200 OK in 279.865Âµs
>[negroni] Started GET /nodes/278bd6b4e16a8e62ef15aaae22e6abc1
>[negroni] Completed 200 OK in 3.200999ms
>[negroni] Started GET /nodes/70423cc0bcd044fe5ba8bbbd256a3e49
>[negroni] Completed 200 OK in 1.764225ms
>[negroni] Started GET /nodes/d942fe6c0ee5691b7cc263968f97b650
>[negroni] Completed 200 OK in 891.442Âµs
>[negroni] Started GET /volumes/a29f96e611e83fe05af97f0b595cb460
>[negroni] Completed 200 OK in 1.235017ms
>[negroni] Started POST /volumes/a29f96e611e83fe05af97f0b595cb460/expand
>[heketi] INFO 2018/06/08 08:37:04 Allocating brick set #0
>[negroni] Completed 202 Accepted in 9.336857ms
>[asynchttp] INFO 2018/06/08 08:37:04 asynchttp.go:288: Started job 5dc1008f9ebb211ba152dd249430d699
>[heketi] INFO 2018/06/08 08:37:04 Started async operation: Expand Volume
>[negroni] Started GET /queue/5dc1008f9ebb211ba152dd249430d699
>[negroni] Completed 200 OK in 100.189Âµs
>[heketi] INFO 2018/06/08 08:37:04 Creating brick 46ec6998de7cdc580ac03be455e5a4a6
>[heketi] INFO 2018/06/08 08:37:04 Creating brick 097f338e8c8aee735f656f0b48d684fa
>[heketi] INFO 2018/06/08 08:37:04 Creating brick 57fb807839bd9ec9be7d29605127df0f
>[kubeexec] DEBUG 2018/06/08 08:37:05 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir -p /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_46ec6998de7cdc580ac03be455e5a4a6
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:37:05 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir -p /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_097f338e8c8aee735f656f0b48d684fa
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:37:05 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir -p /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_57fb807839bd9ec9be7d29605127df0f
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:37:05 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvcreate --autobackup=n --poolmetadatasize 16384K --chunksize 256K --size 3145728K --thin vg_3a4297677881963e3f80124971d50eea/tp_097f338e8c8aee735f656f0b48d684fa --virtualsize 3145728K --name brick_097f338e8c8aee735f656f0b48d684fa
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_097f338e8c8aee735f656f0b48d684fa" created.
>[kubeexec] DEBUG 2018/06/08 08:37:05 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvcreate --autobackup=n --poolmetadatasize 16384K --chunksize 256K --size 3145728K --thin vg_b861af9cc67f5bf8d5a8f9e38f56b14e/tp_57fb807839bd9ec9be7d29605127df0f --virtualsize 3145728K --name brick_57fb807839bd9ec9be7d29605127df0f
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_57fb807839bd9ec9be7d29605127df0f" created.
>[kubeexec] DEBUG 2018/06/08 08:37:05 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvcreate --autobackup=n --poolmetadatasize 16384K --chunksize 256K --size 3145728K --thin vg_9394bc70699b006c5460c9f654cf345f/tp_46ec6998de7cdc580ac03be455e5a4a6 --virtualsize 3145728K --name brick_46ec6998de7cdc580ac03be455e5a4a6
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_46ec6998de7cdc580ac03be455e5a4a6" created.
>[kubeexec] DEBUG 2018/06/08 08:37:05 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_57fb807839bd9ec9be7d29605127df0f
>Result: meta-data=/dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_57fb807839bd9ec9be7d29605127df0f isize=512    agcount=8, agsize=98304 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=786432, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:37:05 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_097f338e8c8aee735f656f0b48d684fa
>Result: meta-data=/dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_097f338e8c8aee735f656f0b48d684fa isize=512    agcount=8, agsize=98304 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=786432, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:37:05 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_46ec6998de7cdc580ac03be455e5a4a6
>Result: meta-data=/dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_46ec6998de7cdc580ac03be455e5a4a6 isize=512    agcount=8, agsize=98304 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=786432, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:37:05 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: awk "BEGIN {print \"/dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_57fb807839bd9ec9be7d29605127df0f /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_57fb807839bd9ec9be7d29605127df0f xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:37:05 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: awk "BEGIN {print \"/dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_097f338e8c8aee735f656f0b48d684fa /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_097f338e8c8aee735f656f0b48d684fa xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:37:05 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: awk "BEGIN {print \"/dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_46ec6998de7cdc580ac03be455e5a4a6 /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_46ec6998de7cdc580ac03be455e5a4a6 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:37:05 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_57fb807839bd9ec9be7d29605127df0f /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_57fb807839bd9ec9be7d29605127df0f
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:37:05 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_46ec6998de7cdc580ac03be455e5a4a6 /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_46ec6998de7cdc580ac03be455e5a4a6
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:37:05 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_097f338e8c8aee735f656f0b48d684fa /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_097f338e8c8aee735f656f0b48d684fa
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:37:05 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_57fb807839bd9ec9be7d29605127df0f/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:37:05 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_097f338e8c8aee735f656f0b48d684fa/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:37:05 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_46ec6998de7cdc580ac03be455e5a4a6/brick
>Result: 
>[negroni] Started GET /queue/5dc1008f9ebb211ba152dd249430d699
>[negroni] Completed 200 OK in 155.51Âµs
>[negroni] Started GET /queue/5dc1008f9ebb211ba152dd249430d699
>[negroni] Completed 200 OK in 228.382Âµs
>[negroni] Started GET /queue/5dc1008f9ebb211ba152dd249430d699
>[negroni] Completed 200 OK in 133.282Âµs
>[negroni] Started GET /queue/5dc1008f9ebb211ba152dd249430d699
>[negroni] Completed 200 OK in 167.744Âµs
>[kubeexec] DEBUG 2018/06/08 08:37:09 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script volume add-brick vol_a29f96e611e83fe05af97f0b595cb460 10.70.47.76:/var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_46ec6998de7cdc580ac03be455e5a4a6/brick 10.70.46.187:/var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_57fb807839bd9ec9be7d29605127df0f/brick 10.70.46.122:/var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_097f338e8c8aee735f656f0b48d684fa/brick
>Result: volume add-brick: success
>[negroni] Started GET /queue/5dc1008f9ebb211ba152dd249430d699
>[negroni] Completed 200 OK in 152.303Âµs
>[negroni] Started GET /queue/5dc1008f9ebb211ba152dd249430d699
>[negroni] Completed 200 OK in 137.811Âµs
>[negroni] Started GET /queue/5dc1008f9ebb211ba152dd249430d699
>[negroni] Completed 200 OK in 132.114Âµs
>[negroni] Started GET /queue/5dc1008f9ebb211ba152dd249430d699
>[negroni] Completed 200 OK in 157.453Âµs
>[negroni] Started GET /queue/5dc1008f9ebb211ba152dd249430d699
>[negroni] Completed 200 OK in 277.868Âµs
>[heketi] INFO 2018/06/08 08:37:14 Starting Node Health Status refresh
>[cmdexec] INFO 2018/06/08 08:37:14 Check Glusterd service status in node dhcp46-187.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 08:37:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:33 UTC; 2h 13min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48fed567_6ae4_11e8_bc4b_005056a5f18a.slice/docker-a506d9aaca4b440d184df40d01e90b2040811c8090bf4c19d892081307b28c0c.scope/system.slice/glusterd.service
>           ââ  433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1228 /usr/sbin/glusterfsd -s 10.70.46.187 --volfile-id heketidbstorage.10.70.46.187.var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.187-var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.pid -S /var/run/gluster/c08898c2e3b0190c62df3a9d4a013c92.socket --brick-name /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.log --xlator-option *-posix.glusterd-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ11007 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/8fb729a0bbd51b204491c603c84acaa4.socket --xlator-option *replicate*.node-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034
>
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Unit glusterd.service entered failed state.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: glusterd.service failed.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:33 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 08:37:14 Periodic health check status: node 278bd6b4e16a8e62ef15aaae22e6abc1 up=true
>[cmdexec] INFO 2018/06/08 08:37:14 Check Glusterd service status in node dhcp46-122.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 08:37:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:42 UTC; 2h 13min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4900c968_6ae4_11e8_bc4b_005056a5f18a.slice/docker-438fb5fb16a82a3e701b28fa1f1ef8623dc37ca06374c604e5781a2b2ab2e06d.scope/system.slice/glusterd.service
>           ââ  433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1106 /usr/sbin/glusterfsd -s 10.70.46.122 --volfile-id heketidbstorage.10.70.46.122.var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.122-var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.pid -S /var/run/gluster/9fa3ac3b12826912287adc88c26ac134.socket --brick-name /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_29e5b36d921cca1b0699871d9727c932/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.log --xlator-option *-posix.glusterd-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ11243 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f8994cdb09351d83e70da08261d99c64.socket --xlator-option *replicate*.node-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0
>
>Jun 08 06:23:40 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:42 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 08:37:14 Periodic health check status: node 70423cc0bcd044fe5ba8bbbd256a3e49 up=true
>[cmdexec] INFO 2018/06/08 08:37:14 Check Glusterd service status in node dhcp47-76.lab.eng.blr.redhat.com
>[negroni] Started GET /queue/5dc1008f9ebb211ba152dd249430d699
>[negroni] Completed 200 OK in 234.812Âµs
>[negroni] Started GET /queue/5dc1008f9ebb211ba152dd249430d699
>[negroni] Completed 200 OK in 225.148Âµs
>[negroni] Started GET /queue/5dc1008f9ebb211ba152dd249430d699
>[negroni] Completed 200 OK in 162.685Âµs
>[negroni] Started GET /queue/5dc1008f9ebb211ba152dd249430d699
>[negroni] Completed 200 OK in 279.348Âµs
>[negroni] Started GET /queue/5dc1008f9ebb211ba152dd249430d699
>[negroni] Completed 200 OK in 230.472Âµs
>[kubeexec] DEBUG 2018/06/08 08:37:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script volume rebalance vol_a29f96e611e83fe05af97f0b595cb460 start
>Result: volume rebalance: vol_a29f96e611e83fe05af97f0b595cb460: success: Rebalance on vol_a29f96e611e83fe05af97f0b595cb460 has been started successfully. Use rebalance status command to check status of the rebalance process.
>ID: a49b6489-335d-4802-b5cb-dd6f098d13e7
>
>[heketi] INFO 2018/06/08 08:37:19 Expand Volume succeeded
>[asynchttp] INFO 2018/06/08 08:37:19 asynchttp.go:292: Completed job 5dc1008f9ebb211ba152dd249430d699 in 14.526564152s
>[kubeexec] DEBUG 2018/06/08 08:37:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 07:25:20 UTC; 1h 11min ago
>  Process: 828 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 830 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5cb2299_6aec_11e8_bc4b_005056a5f18a.slice/docker-7f9223abc9a8641cea68f374f4039235515978ee2dab970ac6cbef182c535426.scope/system.slice/glusterd.service
>           ââ 830 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ1197 /usr/sbin/glusterfsd -s 10.70.47.76 --volfile-id heketidbstorage.10.70.47.76.var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.76-var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.pid -S /var/run/gluster/b6976f9ddc3abc90e1dfa91c35a7c611.socket --brick-name /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_54c06cdf52c6fe1e3a3f8a9d39cde33d/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.log --xlator-option *-posix.glusterd-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ9232 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/ec468453b11caed5259f5e1abcc65b44.socket --xlator-option *replicate*.node-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94
>
>Jun 08 07:25:17 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 07:25:20 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 08:37:19 Periodic health check status: node d942fe6c0ee5691b7cc263968f97b650 up=true
>[heketi] INFO 2018/06/08 08:37:19 Cleaned 0 nodes from health cache
>[negroni] Started GET /queue/5dc1008f9ebb211ba152dd249430d699
>[negroni] Completed 303 See Other in 188.405Âµs
>[negroni] Started GET /volumes/a29f96e611e83fe05af97f0b595cb460
>[negroni] Completed 200 OK in 6.829644ms
>[negroni] Started GET /clusters
>[negroni] Completed 200 OK in 235.685Âµs
>[negroni] Started GET /clusters/0a73c60efdd4673113b668afea101e6d
>[negroni] Completed 200 OK in 356.846Âµs
>[negroni] Started GET /nodes/278bd6b4e16a8e62ef15aaae22e6abc1
>[negroni] Completed 200 OK in 3.29431ms
>[negroni] Started GET /nodes/70423cc0bcd044fe5ba8bbbd256a3e49
>[negroni] Completed 200 OK in 2.58317ms
>[negroni] Started GET /nodes/d942fe6c0ee5691b7cc263968f97b650
>[negroni] Completed 200 OK in 968.916Âµs
>[negroni] Started GET /volumes/a29f96e611e83fe05af97f0b595cb460
>[negroni] Completed 200 OK in 927.28Âµs
>[negroni] Started POST /volumes/a29f96e611e83fe05af97f0b595cb460/expand
>[heketi] INFO 2018/06/08 08:37:23 Allocating brick set #0
>[negroni] Completed 202 Accepted in 10.254939ms
>[asynchttp] INFO 2018/06/08 08:37:23 asynchttp.go:288: Started job 175d13daae1213f905e671b4e3303d57
>[heketi] INFO 2018/06/08 08:37:23 Started async operation: Expand Volume
>[heketi] INFO 2018/06/08 08:37:23 Creating brick 27443c3e5bbd70edaecc2f07a080b837
>[heketi] INFO 2018/06/08 08:37:23 Creating brick fa2fd6ed4afbf5ab55c24a6bec103abb
>[negroni] Started GET /queue/175d13daae1213f905e671b4e3303d57
>[heketi] INFO 2018/06/08 08:37:23 Creating brick 2a96d70db787cec2e5621e5eb6d0fd04
>[negroni] Completed 200 OK in 108.961Âµs
>[kubeexec] DEBUG 2018/06/08 08:37:23 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir -p /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_27443c3e5bbd70edaecc2f07a080b837
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:37:23 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir -p /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_2a96d70db787cec2e5621e5eb6d0fd04
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:37:23 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir -p /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_fa2fd6ed4afbf5ab55c24a6bec103abb
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:37:24 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvcreate --autobackup=n --poolmetadatasize 16384K --chunksize 256K --size 3145728K --thin vg_66a9af9f6bac95f9d8d556a2f14c29d3/tp_2a96d70db787cec2e5621e5eb6d0fd04 --virtualsize 3145728K --name brick_2a96d70db787cec2e5621e5eb6d0fd04
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_2a96d70db787cec2e5621e5eb6d0fd04" created.
>[kubeexec] DEBUG 2018/06/08 08:37:24 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvcreate --autobackup=n --poolmetadatasize 16384K --chunksize 256K --size 3145728K --thin vg_9394bc70699b006c5460c9f654cf345f/tp_27443c3e5bbd70edaecc2f07a080b837 --virtualsize 3145728K --name brick_27443c3e5bbd70edaecc2f07a080b837
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_27443c3e5bbd70edaecc2f07a080b837" created.
>[kubeexec] DEBUG 2018/06/08 08:37:24 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvcreate --autobackup=n --poolmetadatasize 16384K --chunksize 256K --size 3145728K --thin vg_d389f0278a774bd7443a09af960961d8/tp_fa2fd6ed4afbf5ab55c24a6bec103abb --virtualsize 3145728K --name brick_fa2fd6ed4afbf5ab55c24a6bec103abb
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_fa2fd6ed4afbf5ab55c24a6bec103abb" created.
>[kubeexec] DEBUG 2018/06/08 08:37:24 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_2a96d70db787cec2e5621e5eb6d0fd04
>Result: meta-data=/dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_2a96d70db787cec2e5621e5eb6d0fd04 isize=512    agcount=8, agsize=98304 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=786432, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:37:24 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_27443c3e5bbd70edaecc2f07a080b837
>Result: meta-data=/dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_27443c3e5bbd70edaecc2f07a080b837 isize=512    agcount=8, agsize=98304 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=786432, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:37:24 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_fa2fd6ed4afbf5ab55c24a6bec103abb
>Result: meta-data=/dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_fa2fd6ed4afbf5ab55c24a6bec103abb isize=512    agcount=8, agsize=98304 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=786432, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:37:24 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: awk "BEGIN {print \"/dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_27443c3e5bbd70edaecc2f07a080b837 /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_27443c3e5bbd70edaecc2f07a080b837 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:37:24 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: awk "BEGIN {print \"/dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_2a96d70db787cec2e5621e5eb6d0fd04 /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_2a96d70db787cec2e5621e5eb6d0fd04 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:37:24 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: awk "BEGIN {print \"/dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_fa2fd6ed4afbf5ab55c24a6bec103abb /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_fa2fd6ed4afbf5ab55c24a6bec103abb xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:37:24 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_2a96d70db787cec2e5621e5eb6d0fd04 /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_2a96d70db787cec2e5621e5eb6d0fd04
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:37:24 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_27443c3e5bbd70edaecc2f07a080b837 /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_27443c3e5bbd70edaecc2f07a080b837
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:37:24 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_fa2fd6ed4afbf5ab55c24a6bec103abb /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_fa2fd6ed4afbf5ab55c24a6bec103abb
>Result: 
>[negroni] Started GET /queue/175d13daae1213f905e671b4e3303d57
>[negroni] Completed 200 OK in 158.726Âµs
>[kubeexec] DEBUG 2018/06/08 08:37:24 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_2a96d70db787cec2e5621e5eb6d0fd04/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:37:24 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_27443c3e5bbd70edaecc2f07a080b837/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:37:24 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_fa2fd6ed4afbf5ab55c24a6bec103abb/brick
>Result: 
>[negroni] Started GET /queue/175d13daae1213f905e671b4e3303d57
>[negroni] Completed 200 OK in 160.98Âµs
>[negroni] Started GET /queue/175d13daae1213f905e671b4e3303d57
>[negroni] Completed 200 OK in 359.064Âµs
>[negroni] Started GET /queue/175d13daae1213f905e671b4e3303d57
>[negroni] Completed 200 OK in 259.501Âµs
>[kubeexec] DEBUG 2018/06/08 08:37:28 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script volume add-brick vol_a29f96e611e83fe05af97f0b595cb460 10.70.47.76:/var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_27443c3e5bbd70edaecc2f07a080b837/brick 10.70.46.187:/var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_fa2fd6ed4afbf5ab55c24a6bec103abb/brick 10.70.46.122:/var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_2a96d70db787cec2e5621e5eb6d0fd04/brick
>Result: volume add-brick: success
>[negroni] Started GET /queue/175d13daae1213f905e671b4e3303d57
>[negroni] Completed 200 OK in 140.95Âµs
>[negroni] Started GET /queue/175d13daae1213f905e671b4e3303d57
>[negroni] Completed 200 OK in 245.622Âµs
>[negroni] Started GET /queue/175d13daae1213f905e671b4e3303d57
>[negroni] Completed 200 OK in 359.921Âµs
>[negroni] Started GET /queue/175d13daae1213f905e671b4e3303d57
>[negroni] Completed 200 OK in 138.929Âµs
>[negroni] Started GET /queue/175d13daae1213f905e671b4e3303d57
>[negroni] Completed 200 OK in 180.991Âµs
>[negroni] Started GET /queue/175d13daae1213f905e671b4e3303d57
>[negroni] Completed 200 OK in 234.938Âµs
>[negroni] Started GET /queue/175d13daae1213f905e671b4e3303d57
>[negroni] Completed 200 OK in 162.57Âµs
>[negroni] Started GET /queue/175d13daae1213f905e671b4e3303d57
>[negroni] Completed 200 OK in 229.742Âµs
>[negroni] Started GET /queue/175d13daae1213f905e671b4e3303d57
>[negroni] Completed 200 OK in 187.447Âµs
>[negroni] Started GET /queue/175d13daae1213f905e671b4e3303d57
>[negroni] Completed 200 OK in 274.038Âµs
>[kubeexec] DEBUG 2018/06/08 08:37:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script volume rebalance vol_a29f96e611e83fe05af97f0b595cb460 start
>Result: volume rebalance: vol_a29f96e611e83fe05af97f0b595cb460: success: Rebalance on vol_a29f96e611e83fe05af97f0b595cb460 has been started successfully. Use rebalance status command to check status of the rebalance process.
>ID: 46590f33-516f-4a31-984d-e29c7f233067
>
>[heketi] INFO 2018/06/08 08:37:38 Expand Volume succeeded
>[asynchttp] INFO 2018/06/08 08:37:38 asynchttp.go:292: Completed job 175d13daae1213f905e671b4e3303d57 in 14.7851187s
>[negroni] Started GET /queue/175d13daae1213f905e671b4e3303d57
>[negroni] Completed 303 See Other in 257.718Âµs
>[negroni] Started GET /volumes/a29f96e611e83fe05af97f0b595cb460
>[negroni] Completed 200 OK in 5.356655ms
>[negroni] Started GET /clusters
>[negroni] Completed 200 OK in 271.585Âµs
>[negroni] Started GET /clusters/0a73c60efdd4673113b668afea101e6d
>[negroni] Completed 200 OK in 289.32Âµs
>[negroni] Started GET /nodes/278bd6b4e16a8e62ef15aaae22e6abc1
>[negroni] Completed 200 OK in 4.282973ms
>[negroni] Started GET /nodes/70423cc0bcd044fe5ba8bbbd256a3e49
>[negroni] Completed 200 OK in 2.440086ms
>[negroni] Started GET /nodes/d942fe6c0ee5691b7cc263968f97b650
>[negroni] Completed 200 OK in 1.875202ms
>[negroni] Started GET /volumes/a29f96e611e83fe05af97f0b595cb460
>[negroni] Completed 200 OK in 1.571782ms
>[negroni] Started GET /clusters
>[negroni] Completed 200 OK in 159.169Âµs
>[negroni] Started GET /clusters/0a73c60efdd4673113b668afea101e6d
>[negroni] Completed 200 OK in 311.999Âµs
>[negroni] Started GET /nodes/278bd6b4e16a8e62ef15aaae22e6abc1
>[negroni] Completed 200 OK in 1.024889ms
>[negroni] Started GET /nodes/70423cc0bcd044fe5ba8bbbd256a3e49
>[negroni] Completed 200 OK in 1.674015ms
>[negroni] Started GET /nodes/d942fe6c0ee5691b7cc263968f97b650
>[negroni] Completed 200 OK in 1.119936ms
>[negroni] Started DELETE /volumes/a29f96e611e83fe05af97f0b595cb460
>[negroni] Completed 202 Accepted in 11.526063ms
>[asynchttp] INFO 2018/06/08 08:37:42 asynchttp.go:288: Started job 598142190d8886b152c94f68df77e0e2
>[heketi] INFO 2018/06/08 08:37:42 Started async operation: Delete Volume
>[negroni] Started GET /queue/598142190d8886b152c94f68df77e0e2
>[negroni] Completed 200 OK in 153.633Âµs
>[kubeexec] DEBUG 2018/06/08 08:37:42 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: gluster --mode=script snapshot list vol_a29f96e611e83fe05af97f0b595cb460 --xml
>Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
><cliOutput>
>  <opRet>0</opRet>
>  <opErrno>0</opErrno>
>  <opErrstr/>
>  <snapList>
>    <count>0</count>
>  </snapList>
></cliOutput>
>[negroni] Started GET /queue/598142190d8886b152c94f68df77e0e2
>[negroni] Completed 200 OK in 120.55Âµs
>[negroni] Started GET /queue/598142190d8886b152c94f68df77e0e2
>[negroni] Completed 200 OK in 263.018Âµs
>[kubeexec] DEBUG 2018/06/08 08:37:44 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: gluster --mode=script volume stop vol_a29f96e611e83fe05af97f0b595cb460 force
>Result: volume stop: vol_a29f96e611e83fe05af97f0b595cb460: success
>[kubeexec] DEBUG 2018/06/08 08:37:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: gluster --mode=script volume delete vol_a29f96e611e83fe05af97f0b595cb460
>Result: volume delete: vol_a29f96e611e83fe05af97f0b595cb460: success
>[heketi] INFO 2018/06/08 08:37:45 Deleting brick 07da7af9f0e00909886f0c79a0eef485
>[heketi] INFO 2018/06/08 08:37:45 Deleting brick 097f338e8c8aee735f656f0b48d684fa
>[heketi] INFO 2018/06/08 08:37:45 Deleting brick 27443c3e5bbd70edaecc2f07a080b837
>[heketi] INFO 2018/06/08 08:37:45 Deleting brick 2a96d70db787cec2e5621e5eb6d0fd04
>[heketi] INFO 2018/06/08 08:37:45 Deleting brick 46ec6998de7cdc580ac03be455e5a4a6
>[heketi] INFO 2018/06/08 08:37:45 Deleting brick 57fb807839bd9ec9be7d29605127df0f
>[heketi] INFO 2018/06/08 08:37:45 Deleting brick f526234df2e4ac0fa1c80096c3759590
>[heketi] INFO 2018/06/08 08:37:45 Deleting brick 6ba663d12286dbed5c5caa2b3d947f69
>[heketi] INFO 2018/06/08 08:37:45 Deleting brick fa2fd6ed4afbf5ab55c24a6bec103abb
>[kubeexec] DEBUG 2018/06/08 08:37:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mount | grep -w /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_27443c3e5bbd70edaecc2f07a080b837 | cut -d" " -f1
>Result: /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_27443c3e5bbd70edaecc2f07a080b837
>[kubeexec] DEBUG 2018/06/08 08:37:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mount | grep -w /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_097f338e8c8aee735f656f0b48d684fa | cut -d" " -f1
>Result: /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_097f338e8c8aee735f656f0b48d684fa
>[kubeexec] DEBUG 2018/06/08 08:37:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mount | grep -w /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_07da7af9f0e00909886f0c79a0eef485 | cut -d" " -f1
>Result: /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_07da7af9f0e00909886f0c79a0eef485
>[kubeexec] DEBUG 2018/06/08 08:37:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mount | grep -w /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_46ec6998de7cdc580ac03be455e5a4a6 | cut -d" " -f1
>Result: /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_46ec6998de7cdc580ac03be455e5a4a6
>[kubeexec] DEBUG 2018/06/08 08:37:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mount | grep -w /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_2a96d70db787cec2e5621e5eb6d0fd04 | cut -d" " -f1
>Result: /dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_2a96d70db787cec2e5621e5eb6d0fd04
>[negroni] Started GET /queue/598142190d8886b152c94f68df77e0e2
>[negroni] Completed 200 OK in 152.324Âµs
>[kubeexec] DEBUG 2018/06/08 08:37:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mount | grep -w /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_57fb807839bd9ec9be7d29605127df0f | cut -d" " -f1
>Result: /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_57fb807839bd9ec9be7d29605127df0f
>[kubeexec] DEBUG 2018/06/08 08:37:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mount | grep -w /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_6ba663d12286dbed5c5caa2b3d947f69 | cut -d" " -f1
>Result: /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_6ba663d12286dbed5c5caa2b3d947f69
>[negroni] Started GET /queue/598142190d8886b152c94f68df77e0e2
>[negroni] Completed 200 OK in 136.554Âµs
>[kubeexec] DEBUG 2018/06/08 08:37:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mount | grep -w /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_f526234df2e4ac0fa1c80096c3759590 | cut -d" " -f1
>Result: /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_f526234df2e4ac0fa1c80096c3759590
>[kubeexec] DEBUG 2018/06/08 08:37:47 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mount | grep -w /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_fa2fd6ed4afbf5ab55c24a6bec103abb | cut -d" " -f1
>Result: /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_fa2fd6ed4afbf5ab55c24a6bec103abb
>[negroni] Started GET /queue/598142190d8886b152c94f68df77e0e2
>[negroni] Completed 200 OK in 140.488Âµs
>[kubeexec] DEBUG 2018/06/08 08:37:47 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_27443c3e5bbd70edaecc2f07a080b837
>
>Result:   vg_9394bc70699b006c5460c9f654cf345f/tp_27443c3e5bbd70edaecc2f07a080b837
>[kubeexec] DEBUG 2018/06/08 08:37:47 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_097f338e8c8aee735f656f0b48d684fa
>
>Result:   vg_3a4297677881963e3f80124971d50eea/tp_097f338e8c8aee735f656f0b48d684fa
>[negroni] Started GET /queue/598142190d8886b152c94f68df77e0e2
>[negroni] Completed 200 OK in 239.614Âµs
>[kubeexec] DEBUG 2018/06/08 08:37:48 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_07da7af9f0e00909886f0c79a0eef485
>
>Result:   vg_d389f0278a774bd7443a09af960961d8/tp_07da7af9f0e00909886f0c79a0eef485
>[kubeexec] DEBUG 2018/06/08 08:37:48 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_46ec6998de7cdc580ac03be455e5a4a6
>
>Result:   vg_9394bc70699b006c5460c9f654cf345f/tp_46ec6998de7cdc580ac03be455e5a4a6
>[kubeexec] DEBUG 2018/06/08 08:37:49 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_2a96d70db787cec2e5621e5eb6d0fd04
>
>Result:   vg_66a9af9f6bac95f9d8d556a2f14c29d3/tp_2a96d70db787cec2e5621e5eb6d0fd04
>[negroni] Started GET /queue/598142190d8886b152c94f68df77e0e2
>[negroni] Completed 200 OK in 178.101Âµs
>[kubeexec] DEBUG 2018/06/08 08:37:49 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_57fb807839bd9ec9be7d29605127df0f
>
>Result:   vg_b861af9cc67f5bf8d5a8f9e38f56b14e/tp_57fb807839bd9ec9be7d29605127df0f
>[kubeexec] DEBUG 2018/06/08 08:37:50 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_6ba663d12286dbed5c5caa2b3d947f69
>
>Result:   vg_9394bc70699b006c5460c9f654cf345f/tp_6ba663d12286dbed5c5caa2b3d947f69
>[kubeexec] DEBUG 2018/06/08 08:37:50 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_f526234df2e4ac0fa1c80096c3759590
>
>Result:   vg_3a4297677881963e3f80124971d50eea/tp_f526234df2e4ac0fa1c80096c3759590
>[negroni] Started GET /queue/598142190d8886b152c94f68df77e0e2
>[negroni] Completed 200 OK in 249.281Âµs
>[kubeexec] DEBUG 2018/06/08 08:37:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_fa2fd6ed4afbf5ab55c24a6bec103abb
>
>Result:   vg_d389f0278a774bd7443a09af960961d8/tp_fa2fd6ed4afbf5ab55c24a6bec103abb
>[kubeexec] DEBUG 2018/06/08 08:37:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: umount /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_27443c3e5bbd70edaecc2f07a080b837
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:37:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: umount /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_097f338e8c8aee735f656f0b48d684fa
>Result: 
>[negroni] Started GET /queue/598142190d8886b152c94f68df77e0e2
>[negroni] Completed 200 OK in 216.19Âµs
>[kubeexec] DEBUG 2018/06/08 08:37:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: umount /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_07da7af9f0e00909886f0c79a0eef485
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:37:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: umount /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_46ec6998de7cdc580ac03be455e5a4a6
>Result: 
>[negroni] Started GET /queue/598142190d8886b152c94f68df77e0e2
>[negroni] Completed 200 OK in 283.338Âµs
>[kubeexec] DEBUG 2018/06/08 08:37:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: umount /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_2a96d70db787cec2e5621e5eb6d0fd04
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:37:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: umount /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_57fb807839bd9ec9be7d29605127df0f
>Result: 
>[negroni] Started GET /queue/598142190d8886b152c94f68df77e0e2
>[negroni] Completed 200 OK in 160.548Âµs
>[kubeexec] DEBUG 2018/06/08 08:37:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: umount /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_6ba663d12286dbed5c5caa2b3d947f69
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:37:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: umount /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_f526234df2e4ac0fa1c80096c3759590
>Result: 
>[negroni] Started GET /queue/598142190d8886b152c94f68df77e0e2
>[negroni] Completed 200 OK in 172.398Âµs
>[kubeexec] DEBUG 2018/06/08 08:37:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: umount /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_fa2fd6ed4afbf5ab55c24a6bec103abb
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:37:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: sed -i.save "/brick_27443c3e5bbd70edaecc2f07a080b837/d" /var/lib/heketi/fstab
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:37:55 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: sed -i.save "/brick_097f338e8c8aee735f656f0b48d684fa/d" /var/lib/heketi/fstab
>Result: 
>[negroni] Started GET /queue/598142190d8886b152c94f68df77e0e2
>[negroni] Completed 200 OK in 233.938Âµs
>[kubeexec] DEBUG 2018/06/08 08:37:55 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: sed -i.save "/brick_07da7af9f0e00909886f0c79a0eef485/d" /var/lib/heketi/fstab
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:37:55 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: sed -i.save "/brick_46ec6998de7cdc580ac03be455e5a4a6/d" /var/lib/heketi/fstab
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:37:56 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: sed -i.save "/brick_2a96d70db787cec2e5621e5eb6d0fd04/d" /var/lib/heketi/fstab
>Result: 
>[negroni] Started GET /queue/598142190d8886b152c94f68df77e0e2
>[negroni] Completed 200 OK in 226.582Âµs
>[kubeexec] DEBUG 2018/06/08 08:37:57 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: sed -i.save "/brick_57fb807839bd9ec9be7d29605127df0f/d" /var/lib/heketi/fstab
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:37:57 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: sed -i.save "/brick_6ba663d12286dbed5c5caa2b3d947f69/d" /var/lib/heketi/fstab
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:37:57 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: sed -i.save "/brick_f526234df2e4ac0fa1c80096c3759590/d" /var/lib/heketi/fstab
>Result: 
>[negroni] Started GET /queue/598142190d8886b152c94f68df77e0e2
>[negroni] Completed 200 OK in 212.892Âµs
>[kubeexec] DEBUG 2018/06/08 08:37:58 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: sed -i.save "/brick_fa2fd6ed4afbf5ab55c24a6bec103abb/d" /var/lib/heketi/fstab
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:37:58 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvremove --autobackup=n -f /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_27443c3e5bbd70edaecc2f07a080b837
>
>Result:   Logical volume "brick_27443c3e5bbd70edaecc2f07a080b837" successfully removed
>[negroni] Started GET /queue/598142190d8886b152c94f68df77e0e2
>[negroni] Completed 200 OK in 267.708Âµs
>[kubeexec] DEBUG 2018/06/08 08:37:58 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvremove --autobackup=n -f /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_097f338e8c8aee735f656f0b48d684fa
>
>Result:   Logical volume "brick_097f338e8c8aee735f656f0b48d684fa" successfully removed
>[kubeexec] DEBUG 2018/06/08 08:37:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvremove --autobackup=n -f /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_07da7af9f0e00909886f0c79a0eef485
>
>Result:   Logical volume "brick_07da7af9f0e00909886f0c79a0eef485" successfully removed
>[negroni] Started GET /queue/598142190d8886b152c94f68df77e0e2
>[negroni] Completed 200 OK in 192.782Âµs
>[kubeexec] DEBUG 2018/06/08 08:37:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvremove --autobackup=n -f /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_46ec6998de7cdc580ac03be455e5a4a6
>
>Result:   Logical volume "brick_46ec6998de7cdc580ac03be455e5a4a6" successfully removed
>[kubeexec] DEBUG 2018/06/08 08:37:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvremove --autobackup=n -f /dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_2a96d70db787cec2e5621e5eb6d0fd04
>
>Result:   Logical volume "brick_2a96d70db787cec2e5621e5eb6d0fd04" successfully removed
>[negroni] Started GET /queue/598142190d8886b152c94f68df77e0e2
>[negroni] Completed 200 OK in 238.128Âµs
>[kubeexec] DEBUG 2018/06/08 08:38:00 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvremove --autobackup=n -f /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_57fb807839bd9ec9be7d29605127df0f
>
>Result:   Logical volume "brick_57fb807839bd9ec9be7d29605127df0f" successfully removed
>[kubeexec] DEBUG 2018/06/08 08:38:00 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvremove --autobackup=n -f /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_6ba663d12286dbed5c5caa2b3d947f69
>
>Result:   Logical volume "brick_6ba663d12286dbed5c5caa2b3d947f69" successfully removed
>[kubeexec] DEBUG 2018/06/08 08:38:01 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvremove --autobackup=n -f /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_f526234df2e4ac0fa1c80096c3759590
>
>Result:   Logical volume "brick_f526234df2e4ac0fa1c80096c3759590" successfully removed
>[negroni] Started GET /queue/598142190d8886b152c94f68df77e0e2
>[negroni] Completed 200 OK in 152.348Âµs
>[kubeexec] DEBUG 2018/06/08 08:38:01 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvremove --autobackup=n -f /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_fa2fd6ed4afbf5ab55c24a6bec103abb
>
>Result:   Logical volume "brick_fa2fd6ed4afbf5ab55c24a6bec103abb" successfully removed
>[kubeexec] DEBUG 2018/06/08 08:38:02 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvs --noheadings --options=thin_count   vg_9394bc70699b006c5460c9f654cf345f/tp_27443c3e5bbd70edaecc2f07a080b837
>
>Result:        0
>[kubeexec] DEBUG 2018/06/08 08:38:02 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvs --noheadings --options=thin_count   vg_3a4297677881963e3f80124971d50eea/tp_097f338e8c8aee735f656f0b48d684fa
>
>Result:        0
>[negroni] Started GET /queue/598142190d8886b152c94f68df77e0e2
>[negroni] Completed 200 OK in 236.582Âµs
>[kubeexec] DEBUG 2018/06/08 08:38:03 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvs --noheadings --options=thin_count   vg_d389f0278a774bd7443a09af960961d8/tp_07da7af9f0e00909886f0c79a0eef485
>
>Result:        0
>[kubeexec] DEBUG 2018/06/08 08:38:03 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvs --noheadings --options=thin_count   vg_9394bc70699b006c5460c9f654cf345f/tp_46ec6998de7cdc580ac03be455e5a4a6
>
>Result:        0
>[kubeexec] DEBUG 2018/06/08 08:38:03 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvs --noheadings --options=thin_count   vg_66a9af9f6bac95f9d8d556a2f14c29d3/tp_2a96d70db787cec2e5621e5eb6d0fd04
>
>Result:        0
>[negroni] Started GET /queue/598142190d8886b152c94f68df77e0e2
>[negroni] Completed 200 OK in 165.081Âµs
>[kubeexec] DEBUG 2018/06/08 08:38:04 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvs --noheadings --options=thin_count   vg_b861af9cc67f5bf8d5a8f9e38f56b14e/tp_57fb807839bd9ec9be7d29605127df0f
>
>Result:        0
>[kubeexec] DEBUG 2018/06/08 08:38:04 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvs --noheadings --options=thin_count   vg_9394bc70699b006c5460c9f654cf345f/tp_6ba663d12286dbed5c5caa2b3d947f69
>
>Result:        0
>[negroni] Started GET /queue/598142190d8886b152c94f68df77e0e2
>[negroni] Completed 200 OK in 133.581Âµs
>[kubeexec] DEBUG 2018/06/08 08:38:04 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvs --noheadings --options=thin_count   vg_3a4297677881963e3f80124971d50eea/tp_f526234df2e4ac0fa1c80096c3759590
>
>Result:        0
>[kubeexec] DEBUG 2018/06/08 08:38:05 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvs --noheadings --options=thin_count   vg_d389f0278a774bd7443a09af960961d8/tp_fa2fd6ed4afbf5ab55c24a6bec103abb
>
>Result:        0
>[negroni] Started GET /queue/598142190d8886b152c94f68df77e0e2
>[negroni] Completed 200 OK in 212.466Âµs
>[kubeexec] DEBUG 2018/06/08 08:38:05 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvremove --autobackup=n -f   vg_9394bc70699b006c5460c9f654cf345f/tp_27443c3e5bbd70edaecc2f07a080b837
>
>Result:   Logical volume "tp_27443c3e5bbd70edaecc2f07a080b837" successfully removed
>[kubeexec] DEBUG 2018/06/08 08:38:05 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvremove --autobackup=n -f   vg_3a4297677881963e3f80124971d50eea/tp_097f338e8c8aee735f656f0b48d684fa
>
>Result:   Logical volume "tp_097f338e8c8aee735f656f0b48d684fa" successfully removed
>[negroni] Started GET /queue/598142190d8886b152c94f68df77e0e2
>[negroni] Completed 200 OK in 216.788Âµs
>[kubeexec] DEBUG 2018/06/08 08:38:06 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvremove --autobackup=n -f   vg_d389f0278a774bd7443a09af960961d8/tp_07da7af9f0e00909886f0c79a0eef485
>
>Result:   Logical volume "tp_07da7af9f0e00909886f0c79a0eef485" successfully removed
>[kubeexec] DEBUG 2018/06/08 08:38:06 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvremove --autobackup=n -f   vg_9394bc70699b006c5460c9f654cf345f/tp_46ec6998de7cdc580ac03be455e5a4a6
>
>Result:   Logical volume "tp_46ec6998de7cdc580ac03be455e5a4a6" successfully removed
>[kubeexec] DEBUG 2018/06/08 08:38:07 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvremove --autobackup=n -f   vg_66a9af9f6bac95f9d8d556a2f14c29d3/tp_2a96d70db787cec2e5621e5eb6d0fd04
>
>Result:   Logical volume "tp_2a96d70db787cec2e5621e5eb6d0fd04" successfully removed
>[negroni] Started GET /queue/598142190d8886b152c94f68df77e0e2
>[negroni] Completed 200 OK in 154.068Âµs
>[kubeexec] DEBUG 2018/06/08 08:38:07 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvremove --autobackup=n -f   vg_b861af9cc67f5bf8d5a8f9e38f56b14e/tp_57fb807839bd9ec9be7d29605127df0f
>
>Result:   Logical volume "tp_57fb807839bd9ec9be7d29605127df0f" successfully removed
>[kubeexec] DEBUG 2018/06/08 08:38:08 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvremove --autobackup=n -f   vg_9394bc70699b006c5460c9f654cf345f/tp_6ba663d12286dbed5c5caa2b3d947f69
>
>Result:   Logical volume "tp_6ba663d12286dbed5c5caa2b3d947f69" successfully removed
>[kubeexec] DEBUG 2018/06/08 08:38:08 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvremove --autobackup=n -f   vg_3a4297677881963e3f80124971d50eea/tp_f526234df2e4ac0fa1c80096c3759590
>
>Result:   Logical volume "tp_f526234df2e4ac0fa1c80096c3759590" successfully removed
>[negroni] Started GET /queue/598142190d8886b152c94f68df77e0e2
>[negroni] Completed 200 OK in 198.451Âµs
>[kubeexec] DEBUG 2018/06/08 08:38:08 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvremove --autobackup=n -f   vg_d389f0278a774bd7443a09af960961d8/tp_fa2fd6ed4afbf5ab55c24a6bec103abb
>
>Result:   Logical volume "tp_fa2fd6ed4afbf5ab55c24a6bec103abb" successfully removed
>[kubeexec] DEBUG 2018/06/08 08:38:09 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: rmdir /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_27443c3e5bbd70edaecc2f07a080b837
>Result: 
>[negroni] Started GET /queue/598142190d8886b152c94f68df77e0e2
>[negroni] Completed 200 OK in 142.655Âµs
>[kubeexec] DEBUG 2018/06/08 08:38:09 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: rmdir /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_097f338e8c8aee735f656f0b48d684fa
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:38:09 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: rmdir /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_07da7af9f0e00909886f0c79a0eef485
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:38:10 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: rmdir /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_46ec6998de7cdc580ac03be455e5a4a6
>Result: 
>[negroni] Started GET /queue/598142190d8886b152c94f68df77e0e2
>[negroni] Completed 200 OK in 205.889Âµs
>[kubeexec] DEBUG 2018/06/08 08:38:10 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: rmdir /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_2a96d70db787cec2e5621e5eb6d0fd04
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:38:11 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: rmdir /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_57fb807839bd9ec9be7d29605127df0f
>Result: 
>[negroni] Started GET /queue/598142190d8886b152c94f68df77e0e2
>[negroni] Completed 200 OK in 134.529Âµs
>[kubeexec] DEBUG 2018/06/08 08:38:11 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: rmdir /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_6ba663d12286dbed5c5caa2b3d947f69
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:38:12 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: rmdir /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_f526234df2e4ac0fa1c80096c3759590
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:38:12 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: rmdir /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_fa2fd6ed4afbf5ab55c24a6bec103abb
>Result: 
>[heketi] INFO 2018/06/08 08:38:12 Delete Volume succeeded
>[asynchttp] INFO 2018/06/08 08:38:12 asynchttp.go:292: Completed job 598142190d8886b152c94f68df77e0e2 in 29.804664552s
>[negroni] Started GET /queue/598142190d8886b152c94f68df77e0e2
>[negroni] Completed 204 No Content in 129.935Âµs
>[negroni] Started GET /clusters
>[negroni] Completed 200 OK in 2.472404ms
>[negroni] Started GET /clusters/0a73c60efdd4673113b668afea101e6d
>[negroni] Completed 200 OK in 297.399Âµs
>[negroni] Started GET /nodes/278bd6b4e16a8e62ef15aaae22e6abc1
>[negroni] Completed 200 OK in 3.548846ms
>[negroni] Started GET /nodes/70423cc0bcd044fe5ba8bbbd256a3e49
>[negroni] Completed 200 OK in 2.061511ms
>[negroni] Started GET /nodes/d942fe6c0ee5691b7cc263968f97b650
>[negroni] Completed 200 OK in 1.529223ms
>[negroni] Started GET /clusters
>[negroni] Completed 200 OK in 223.305Âµs
>[negroni] Started GET /clusters/0a73c60efdd4673113b668afea101e6d
>[negroni] Completed 200 OK in 274.892Âµs
>[negroni] Started GET /nodes/278bd6b4e16a8e62ef15aaae22e6abc1
>[negroni] Completed 200 OK in 834.907Âµs
>[negroni] Started POST /devices
>[heketi] INFO 2018/06/08 08:38:13 Adding device /dev/sdf to node 278bd6b4e16a8e62ef15aaae22e6abc1
>[asynchttp] INFO 2018/06/08 08:38:13 asynchttp.go:288: Started job 1cea9b6fd5211a402ce922cc12ee59a1
>[negroni] Completed 202 Accepted in 11.105243ms
>[negroni] Started GET /queue/1cea9b6fd5211a402ce922cc12ee59a1
>[negroni] Completed 200 OK in 155.914Âµs
>[kubeexec] DEBUG 2018/06/08 08:38:13 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: pvcreate --metadatasize=128M --dataalignment=256K '/dev/sdf'
>Result:   Physical volume "/dev/sdf" successfully created.
>[kubeexec] DEBUG 2018/06/08 08:38:13 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: vgcreate --autobackup=n vg_6ecedef27ecea4a51dc0dbaeb83f2328 /dev/sdf
>Result:   Volume group "vg_6ecedef27ecea4a51dc0dbaeb83f2328" successfully created
>[kubeexec] DEBUG 2018/06/08 08:38:13 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: vgdisplay -c vg_6ecedef27ecea4a51dc0dbaeb83f2328
>Result:   vg_6ecedef27ecea4a51dc0dbaeb83f2328:r/w:772:-1:0:0:0:-1:0:1:1:104722432:4096:25567:0:25567:JbrxAE-Ue84-dyos-Lxwf-CTny-1zfk-WQtP80
>[cmdexec] DEBUG 2018/06/08 08:38:13 /src/github.com/heketi/heketi/executors/cmdexec/device.go:147: Size of /dev/sdf in dhcp46-187.lab.eng.blr.redhat.com is 104722432
>[heketi] INFO 2018/06/08 08:38:13 Added device /dev/sdf
>[asynchttp] INFO 2018/06/08 08:38:13 asynchttp.go:292: Completed job 1cea9b6fd5211a402ce922cc12ee59a1 in 519.180802ms
>[negroni] Started GET /queue/1cea9b6fd5211a402ce922cc12ee59a1
>[negroni] Completed 204 No Content in 256.471Âµs
>[negroni] Started GET /nodes/70423cc0bcd044fe5ba8bbbd256a3e49
>[negroni] Completed 200 OK in 6.897718ms
>[negroni] Started POST /devices
>[heketi] INFO 2018/06/08 08:38:14 Adding device /dev/sdf to node 70423cc0bcd044fe5ba8bbbd256a3e49
>[negroni] Completed 202 Accepted in 10.483709ms
>[asynchttp] INFO 2018/06/08 08:38:14 asynchttp.go:288: Started job 609c4a963033c593d3ce23492977a6ab
>[negroni] Started GET /queue/609c4a963033c593d3ce23492977a6ab
>[negroni] Completed 200 OK in 126.494Âµs
>[kubeexec] DEBUG 2018/06/08 08:38:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: pvcreate --metadatasize=128M --dataalignment=256K '/dev/sdf'
>Result:   Physical volume "/dev/sdf" successfully created.
>[kubeexec] DEBUG 2018/06/08 08:38:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: vgcreate --autobackup=n vg_c7076a74d8fd3bf1c2dcfc77b8d01ea1 /dev/sdf
>Result:   Volume group "vg_c7076a74d8fd3bf1c2dcfc77b8d01ea1" successfully created
>[kubeexec] DEBUG 2018/06/08 08:38:15 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: vgdisplay -c vg_c7076a74d8fd3bf1c2dcfc77b8d01ea1
>Result:   vg_c7076a74d8fd3bf1c2dcfc77b8d01ea1:r/w:772:-1:0:0:0:-1:0:1:1:104722432:4096:25567:0:25567:bMTslf-fczU-xI8F-esVf-351y-FZEC-thqWUx
>[cmdexec] DEBUG 2018/06/08 08:38:15 /src/github.com/heketi/heketi/executors/cmdexec/device.go:147: Size of /dev/sdf in dhcp46-122.lab.eng.blr.redhat.com is 104722432
>[heketi] INFO 2018/06/08 08:38:15 Added device /dev/sdf
>[asynchttp] INFO 2018/06/08 08:38:15 asynchttp.go:292: Completed job 609c4a963033c593d3ce23492977a6ab in 592.10989ms
>[negroni] Started GET /queue/609c4a963033c593d3ce23492977a6ab
>[negroni] Completed 204 No Content in 189.55Âµs
>[negroni] Started GET /nodes/d942fe6c0ee5691b7cc263968f97b650
>[negroni] Completed 200 OK in 4.938804ms
>[negroni] Started POST /devices
>[heketi] INFO 2018/06/08 08:38:15 Adding device /dev/sdf to node d942fe6c0ee5691b7cc263968f97b650
>[negroni] Completed 202 Accepted in 8.246058ms
>[asynchttp] INFO 2018/06/08 08:38:15 asynchttp.go:288: Started job 68a9e009b9ea3887ddbadce2e33271a4
>[negroni] Started GET /queue/68a9e009b9ea3887ddbadce2e33271a4
>[negroni] Completed 200 OK in 128.082Âµs
>[kubeexec] DEBUG 2018/06/08 08:38:15 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: pvcreate --metadatasize=128M --dataalignment=256K '/dev/sdf'
>Result:   Physical volume "/dev/sdf" successfully created.
>[kubeexec] DEBUG 2018/06/08 08:38:16 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: vgcreate --autobackup=n vg_624b4458a2f2db51030873f216ff446a /dev/sdf
>Result:   Volume group "vg_624b4458a2f2db51030873f216ff446a" successfully created
>[kubeexec] DEBUG 2018/06/08 08:38:16 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: vgdisplay -c vg_624b4458a2f2db51030873f216ff446a
>Result:   vg_624b4458a2f2db51030873f216ff446a:r/w:772:-1:0:0:0:-1:0:1:1:104722432:4096:25567:0:25567:Ybuqv7-pPcl-5ROM-QSLa-vC1Q-zAQR-Tk5C43
>[cmdexec] DEBUG 2018/06/08 08:38:16 /src/github.com/heketi/heketi/executors/cmdexec/device.go:147: Size of /dev/sdf in dhcp47-76.lab.eng.blr.redhat.com is 104722432
>[heketi] INFO 2018/06/08 08:38:16 Added device /dev/sdf
>[asynchttp] INFO 2018/06/08 08:38:16 asynchttp.go:292: Completed job 68a9e009b9ea3887ddbadce2e33271a4 in 523.793649ms
>[negroni] Started GET /queue/68a9e009b9ea3887ddbadce2e33271a4
>[negroni] Completed 204 No Content in 133.618Âµs
>[negroni] Started GET /nodes/d942fe6c0ee5691b7cc263968f97b650
>[negroni] Completed 200 OK in 4.578965ms
>[negroni] Started GET /nodes/70423cc0bcd044fe5ba8bbbd256a3e49
>[negroni] Completed 200 OK in 2.100891ms
>[negroni] Started GET /nodes/278bd6b4e16a8e62ef15aaae22e6abc1
>[negroni] Completed 200 OK in 906.679Âµs
>[negroni] Started GET /nodes/d942fe6c0ee5691b7cc263968f97b650
>[negroni] Completed 200 OK in 1.443692ms
>[negroni] Started GET /nodes/70423cc0bcd044fe5ba8bbbd256a3e49
>[negroni] Completed 200 OK in 1.480624ms
>[negroni] Started GET /nodes/278bd6b4e16a8e62ef15aaae22e6abc1
>[negroni] Completed 200 OK in 1.163396ms
>[negroni] Started GET /nodes/d942fe6c0ee5691b7cc263968f97b650
>[negroni] Completed 200 OK in 939.89Âµs
>[negroni] Started POST /devices/624b4458a2f2db51030873f216ff446a/state
>[negroni] Completed 202 Accepted in 408.501Âµs
>[asynchttp] INFO 2018/06/08 08:38:17 asynchttp.go:288: Started job 1a86a861f6d7007cdd7d7a31bc160dcf
>[negroni] Started GET /queue/1a86a861f6d7007cdd7d7a31bc160dcf
>[negroni] Completed 200 OK in 111.302Âµs
>[asynchttp] INFO 2018/06/08 08:38:17 asynchttp.go:292: Completed job 1a86a861f6d7007cdd7d7a31bc160dcf in 8.583594ms
>[negroni] Started GET /queue/1a86a861f6d7007cdd7d7a31bc160dcf
>[negroni] Completed 204 No Content in 228.808Âµs
>[negroni] Started GET /nodes/70423cc0bcd044fe5ba8bbbd256a3e49
>[negroni] Completed 200 OK in 7.588813ms
>[negroni] Started POST /devices/c7076a74d8fd3bf1c2dcfc77b8d01ea1/state
>[negroni] Completed 202 Accepted in 607.292Âµs
>[asynchttp] INFO 2018/06/08 08:38:18 asynchttp.go:288: Started job 05c736ee53189ef6a0e5643a96855daa
>[negroni] Started GET /queue/05c736ee53189ef6a0e5643a96855daa
>[negroni] Completed 200 OK in 89.161Âµs
>[asynchttp] INFO 2018/06/08 08:38:18 asynchttp.go:292: Completed job 05c736ee53189ef6a0e5643a96855daa in 9.936291ms
>[negroni] Started GET /queue/05c736ee53189ef6a0e5643a96855daa
>[negroni] Completed 204 No Content in 169.757Âµs
>[negroni] Started GET /nodes/278bd6b4e16a8e62ef15aaae22e6abc1
>[negroni] Completed 200 OK in 5.178057ms
>[negroni] Started POST /devices/6ecedef27ecea4a51dc0dbaeb83f2328/state
>[negroni] Completed 202 Accepted in 423.681Âµs
>[asynchttp] INFO 2018/06/08 08:38:20 asynchttp.go:288: Started job 3d537edd99cd8b0677af33799374ed9f
>[negroni] Started GET /queue/3d537edd99cd8b0677af33799374ed9f
>[negroni] Completed 200 OK in 121.402Âµs
>[asynchttp] INFO 2018/06/08 08:38:20 asynchttp.go:292: Completed job 3d537edd99cd8b0677af33799374ed9f in 7.126426ms
>[negroni] Started GET /queue/3d537edd99cd8b0677af33799374ed9f
>[negroni] Completed 204 No Content in 150.491Âµs
>[negroni] Started POST /volumes
>[heketi] INFO 2018/06/08 08:38:21 Allocating brick set #0
>[heketi] INFO 2018/06/08 08:38:21 Allocating brick set #0
>[heketi] INFO 2018/06/08 08:38:21 Allocating brick set #1
>[heketi] INFO 2018/06/08 08:38:21 Allocating brick set #0
>[heketi] INFO 2018/06/08 08:38:21 Allocating brick set #1
>[heketi] INFO 2018/06/08 08:38:21 Allocating brick set #2
>[heketi] INFO 2018/06/08 08:38:21 Allocating brick set #3
>[heketi] INFO 2018/06/08 08:38:21 Allocating brick set #0
>[heketi] INFO 2018/06/08 08:38:21 Allocating brick set #1
>[heketi] INFO 2018/06/08 08:38:21 Allocating brick set #2
>[heketi] INFO 2018/06/08 08:38:21 Allocating brick set #3
>[heketi] INFO 2018/06/08 08:38:21 Allocating brick set #4
>[heketi] INFO 2018/06/08 08:38:21 Allocating brick set #5
>[heketi] INFO 2018/06/08 08:38:21 Allocating brick set #6
>[heketi] ERROR 2018/06/08 08:38:21 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:1167: Create Volume Build Failed: No space
>[negroni] Completed 500 Internal Server Error in 23.548071ms
>[negroni] Started POST /devices/624b4458a2f2db51030873f216ff446a/state
>[negroni] Completed 202 Accepted in 634.885Âµs
>[asynchttp] INFO 2018/06/08 08:38:21 asynchttp.go:288: Started job b426ac62da12f030d840bac94e0689a1
>[asynchttp] INFO 2018/06/08 08:38:21 asynchttp.go:292: Completed job b426ac62da12f030d840bac94e0689a1 in 1.523Âµs
>[negroni] Started GET /queue/b426ac62da12f030d840bac94e0689a1
>[negroni] Completed 204 No Content in 96.763Âµs
>[negroni] Started POST /devices/624b4458a2f2db51030873f216ff446a/state
>[negroni] Completed 202 Accepted in 592.459Âµs
>[asynchttp] INFO 2018/06/08 08:38:21 asynchttp.go:288: Started job 92b3085a0d6c33e8d55baad007a33eea
>[heketi] INFO 2018/06/08 08:38:21 Running Remove Device
>[negroni] Started GET /queue/92b3085a0d6c33e8d55baad007a33eea
>[negroni] Completed 200 OK in 156.031Âµs
>[asynchttp] INFO 2018/06/08 08:38:21 asynchttp.go:292: Completed job 92b3085a0d6c33e8d55baad007a33eea in 13.197973ms
>[negroni] Started GET /queue/92b3085a0d6c33e8d55baad007a33eea
>[negroni] Completed 204 No Content in 141.439Âµs
>[negroni] Started DELETE /devices/624b4458a2f2db51030873f216ff446a
>[heketi] INFO 2018/06/08 08:38:22 Deleting device 624b4458a2f2db51030873f216ff446a on node d942fe6c0ee5691b7cc263968f97b650
>[negroni] Completed 202 Accepted in 1.197344ms
>[asynchttp] INFO 2018/06/08 08:38:22 asynchttp.go:288: Started job 6b4b160437a9518ab20c9d2c78893580
>[negroni] Started GET /queue/6b4b160437a9518ab20c9d2c78893580
>[negroni] Completed 200 OK in 89.561Âµs
>[kubeexec] DEBUG 2018/06/08 08:38:22 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: vgremove vg_624b4458a2f2db51030873f216ff446a
>Result:   Volume group "vg_624b4458a2f2db51030873f216ff446a" successfully removed
>[kubeexec] DEBUG 2018/06/08 08:38:22 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: pvremove '/dev/sdf'
>Result:   Labels on physical volume "/dev/sdf" successfully wiped.
>[kubeexec] ERROR 2018/06/08 08:38:23 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [ls /var/lib/heketi/mounts/vg_624b4458a2f2db51030873f216ff446a] on glusterfs-storage-gxp7c: Err[command terminated with exit code 2]: Stdout []: Stderr [ls: cannot access /var/lib/heketi/mounts/vg_624b4458a2f2db51030873f216ff446a: No such file or directory
>]
>[heketi] INFO 2018/06/08 08:38:23 Deleted node [624b4458a2f2db51030873f216ff446a]
>[asynchttp] INFO 2018/06/08 08:38:23 asynchttp.go:292: Completed job 6b4b160437a9518ab20c9d2c78893580 in 481.428176ms
>[negroni] Started GET /queue/6b4b160437a9518ab20c9d2c78893580
>[negroni] Completed 204 No Content in 138.896Âµs
>[negroni] Started POST /devices/c7076a74d8fd3bf1c2dcfc77b8d01ea1/state
>[negroni] Completed 202 Accepted in 4.146682ms
>[asynchttp] INFO 2018/06/08 08:38:23 asynchttp.go:288: Started job d6875f6fe15b0c1446f8debfa6be6bdf
>[asynchttp] INFO 2018/06/08 08:38:23 asynchttp.go:292: Completed job d6875f6fe15b0c1446f8debfa6be6bdf in 1.353Âµs
>[negroni] Started GET /queue/d6875f6fe15b0c1446f8debfa6be6bdf
>[negroni] Completed 204 No Content in 189.901Âµs
>[negroni] Started POST /devices/c7076a74d8fd3bf1c2dcfc77b8d01ea1/state
>[negroni] Completed 202 Accepted in 814.794Âµs
>[asynchttp] INFO 2018/06/08 08:38:23 asynchttp.go:288: Started job 6aeeac5db08de10d8a39ffe690f1922e
>[heketi] INFO 2018/06/08 08:38:23 Running Remove Device
>[negroni] Started GET /queue/6aeeac5db08de10d8a39ffe690f1922e
>[negroni] Completed 200 OK in 234.338Âµs
>[asynchttp] INFO 2018/06/08 08:38:23 asynchttp.go:292: Completed job 6aeeac5db08de10d8a39ffe690f1922e in 13.033451ms
>[negroni] Started GET /queue/6aeeac5db08de10d8a39ffe690f1922e
>[negroni] Completed 204 No Content in 182.614Âµs
>[negroni] Started DELETE /devices/c7076a74d8fd3bf1c2dcfc77b8d01ea1
>[heketi] INFO 2018/06/08 08:38:24 Deleting device c7076a74d8fd3bf1c2dcfc77b8d01ea1 on node 70423cc0bcd044fe5ba8bbbd256a3e49
>[negroni] Completed 202 Accepted in 1.586374ms
>[asynchttp] INFO 2018/06/08 08:38:24 asynchttp.go:288: Started job a4d1b58b4a5fd2efe6dc96b8e1fd6fb7
>[negroni] Started GET /queue/a4d1b58b4a5fd2efe6dc96b8e1fd6fb7
>[negroni] Completed 200 OK in 181.922Âµs
>[kubeexec] DEBUG 2018/06/08 08:38:25 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: vgremove vg_c7076a74d8fd3bf1c2dcfc77b8d01ea1
>Result:   Volume group "vg_c7076a74d8fd3bf1c2dcfc77b8d01ea1" successfully removed
>[kubeexec] DEBUG 2018/06/08 08:38:25 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: pvremove '/dev/sdf'
>Result:   Labels on physical volume "/dev/sdf" successfully wiped.
>[kubeexec] ERROR 2018/06/08 08:38:25 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [ls /var/lib/heketi/mounts/vg_c7076a74d8fd3bf1c2dcfc77b8d01ea1] on glusterfs-storage-pg4xc: Err[command terminated with exit code 2]: Stdout []: Stderr [ls: cannot access /var/lib/heketi/mounts/vg_c7076a74d8fd3bf1c2dcfc77b8d01ea1: No such file or directory
>]
>[heketi] INFO 2018/06/08 08:38:25 Deleted node [c7076a74d8fd3bf1c2dcfc77b8d01ea1]
>[asynchttp] INFO 2018/06/08 08:38:25 asynchttp.go:292: Completed job a4d1b58b4a5fd2efe6dc96b8e1fd6fb7 in 479.657523ms
>[negroni] Started GET /queue/a4d1b58b4a5fd2efe6dc96b8e1fd6fb7
>[negroni] Completed 204 No Content in 130.335Âµs
>[negroni] Started POST /devices/6ecedef27ecea4a51dc0dbaeb83f2328/state
>[negroni] Completed 202 Accepted in 3.167729ms
>[asynchttp] INFO 2018/06/08 08:38:26 asynchttp.go:288: Started job 563dd17fee8b606b6686492f966eed4f
>[asynchttp] INFO 2018/06/08 08:38:26 asynchttp.go:292: Completed job 563dd17fee8b606b6686492f966eed4f in 1.659Âµs
>[negroni] Started GET /queue/563dd17fee8b606b6686492f966eed4f
>[negroni] Completed 204 No Content in 128.807Âµs
>[negroni] Started POST /devices/6ecedef27ecea4a51dc0dbaeb83f2328/state
>[negroni] Completed 202 Accepted in 668.542Âµs
>[asynchttp] INFO 2018/06/08 08:38:26 asynchttp.go:288: Started job c982581f2804fbd634da05c14e3b47e8
>[heketi] INFO 2018/06/08 08:38:26 Running Remove Device
>[negroni] Started GET /queue/c982581f2804fbd634da05c14e3b47e8
>[negroni] Completed 200 OK in 147.166Âµs
>[asynchttp] INFO 2018/06/08 08:38:26 asynchttp.go:292: Completed job c982581f2804fbd634da05c14e3b47e8 in 13.567465ms
>[negroni] Started GET /queue/c982581f2804fbd634da05c14e3b47e8
>[negroni] Completed 204 No Content in 232.019Âµs
>[negroni] Started DELETE /devices/6ecedef27ecea4a51dc0dbaeb83f2328
>[heketi] INFO 2018/06/08 08:38:27 Deleting device 6ecedef27ecea4a51dc0dbaeb83f2328 on node 278bd6b4e16a8e62ef15aaae22e6abc1
>[negroni] Completed 202 Accepted in 1.63327ms
>[asynchttp] INFO 2018/06/08 08:38:27 asynchttp.go:288: Started job 71a435fc72b367a50501bf2ebd9af6cf
>[negroni] Started GET /queue/71a435fc72b367a50501bf2ebd9af6cf
>[negroni] Completed 200 OK in 252.281Âµs
>[kubeexec] DEBUG 2018/06/08 08:38:27 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: vgremove vg_6ecedef27ecea4a51dc0dbaeb83f2328
>Result:   Volume group "vg_6ecedef27ecea4a51dc0dbaeb83f2328" successfully removed
>[kubeexec] DEBUG 2018/06/08 08:38:27 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: pvremove '/dev/sdf'
>Result:   Labels on physical volume "/dev/sdf" successfully wiped.
>[kubeexec] ERROR 2018/06/08 08:38:27 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [ls /var/lib/heketi/mounts/vg_6ecedef27ecea4a51dc0dbaeb83f2328] on glusterfs-storage-vsh2m: Err[command terminated with exit code 2]: Stdout []: Stderr [ls: cannot access /var/lib/heketi/mounts/vg_6ecedef27ecea4a51dc0dbaeb83f2328: No such file or directory
>]
>[heketi] INFO 2018/06/08 08:38:27 Deleted node [6ecedef27ecea4a51dc0dbaeb83f2328]
>[asynchttp] INFO 2018/06/08 08:38:27 asynchttp.go:292: Completed job 71a435fc72b367a50501bf2ebd9af6cf in 525.934022ms
>[negroni] Started GET /queue/71a435fc72b367a50501bf2ebd9af6cf
>[negroni] Completed 204 No Content in 176.575Âµs
>[negroni] Started POST /volumes
>[heketi] INFO 2018/06/08 08:38:28 Allocating brick set #0
>[negroni] Completed 202 Accepted in 19.088005ms
>[asynchttp] INFO 2018/06/08 08:38:28 asynchttp.go:288: Started job 065ad33878d141c01bb83ce000d7984d
>[heketi] INFO 2018/06/08 08:38:28 Started async operation: Create Volume
>[negroni] Started GET /queue/065ad33878d141c01bb83ce000d7984d
>[negroni] Completed 200 OK in 116.427Âµs
>[heketi] INFO 2018/06/08 08:38:28 Creating brick 88c50c0f97424a3116d7ecf9870b0f3e
>[heketi] INFO 2018/06/08 08:38:28 Creating brick e106612b2d23de31c6b364829b33a4a9
>[heketi] INFO 2018/06/08 08:38:28 Creating brick de28e8f1db02df454626ab7304cd17b6
>[kubeexec] DEBUG 2018/06/08 08:38:28 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir -p /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_de28e8f1db02df454626ab7304cd17b6
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:38:28 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir -p /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_88c50c0f97424a3116d7ecf9870b0f3e
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:38:28 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir -p /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_e106612b2d23de31c6b364829b33a4a9
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:38:29 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvcreate --autobackup=n --poolmetadatasize 53248K --chunksize 256K --size 10485760K --thin vg_d389f0278a774bd7443a09af960961d8/tp_88c50c0f97424a3116d7ecf9870b0f3e --virtualsize 10485760K --name brick_88c50c0f97424a3116d7ecf9870b0f3e
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_88c50c0f97424a3116d7ecf9870b0f3e" created.
>[kubeexec] DEBUG 2018/06/08 08:38:29 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvcreate --autobackup=n --poolmetadatasize 53248K --chunksize 256K --size 10485760K --thin vg_66a9af9f6bac95f9d8d556a2f14c29d3/tp_de28e8f1db02df454626ab7304cd17b6 --virtualsize 10485760K --name brick_de28e8f1db02df454626ab7304cd17b6
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_de28e8f1db02df454626ab7304cd17b6" created.
>[kubeexec] DEBUG 2018/06/08 08:38:29 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvcreate --autobackup=n --poolmetadatasize 53248K --chunksize 256K --size 10485760K --thin vg_96f1667f2f1ced2c5ef94772922be93b/tp_e106612b2d23de31c6b364829b33a4a9 --virtualsize 10485760K --name brick_e106612b2d23de31c6b364829b33a4a9
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_e106612b2d23de31c6b364829b33a4a9" created.
>[kubeexec] DEBUG 2018/06/08 08:38:29 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_e106612b2d23de31c6b364829b33a4a9
>Result: meta-data=/dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_e106612b2d23de31c6b364829b33a4a9 isize=512    agcount=16, agsize=163840 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=2621440, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:38:29 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_de28e8f1db02df454626ab7304cd17b6
>Result: meta-data=/dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_de28e8f1db02df454626ab7304cd17b6 isize=512    agcount=16, agsize=163840 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=2621440, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:38:29 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_88c50c0f97424a3116d7ecf9870b0f3e
>Result: meta-data=/dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_88c50c0f97424a3116d7ecf9870b0f3e isize=512    agcount=16, agsize=163840 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=2621440, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:38:29 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: awk "BEGIN {print \"/dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_e106612b2d23de31c6b364829b33a4a9 /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_e106612b2d23de31c6b364829b33a4a9 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:38:29 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: awk "BEGIN {print \"/dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_de28e8f1db02df454626ab7304cd17b6 /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_de28e8f1db02df454626ab7304cd17b6 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:38:29 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: awk "BEGIN {print \"/dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_88c50c0f97424a3116d7ecf9870b0f3e /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_88c50c0f97424a3116d7ecf9870b0f3e xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:38:29 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_e106612b2d23de31c6b364829b33a4a9 /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_e106612b2d23de31c6b364829b33a4a9
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:38:29 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_88c50c0f97424a3116d7ecf9870b0f3e /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_88c50c0f97424a3116d7ecf9870b0f3e
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:38:29 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_de28e8f1db02df454626ab7304cd17b6 /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_de28e8f1db02df454626ab7304cd17b6
>Result: 
>[negroni] Started GET /queue/065ad33878d141c01bb83ce000d7984d
>[negroni] Completed 200 OK in 127.976Âµs
>[kubeexec] DEBUG 2018/06/08 08:38:29 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_e106612b2d23de31c6b364829b33a4a9/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:38:29 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_de28e8f1db02df454626ab7304cd17b6/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:38:29 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_88c50c0f97424a3116d7ecf9870b0f3e/brick
>Result: 
>[cmdexec] INFO 2018/06/08 08:38:29 Creating volume vol_4c71beb63ff6abb18f5ad259cbb0a53a replica 3
>[kubeexec] DEBUG 2018/06/08 08:38:29 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: gluster --mode=script volume create vol_4c71beb63ff6abb18f5ad259cbb0a53a replica 3 10.70.46.187:/var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_88c50c0f97424a3116d7ecf9870b0f3e/brick 10.70.46.122:/var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_de28e8f1db02df454626ab7304cd17b6/brick 10.70.47.76:/var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_e106612b2d23de31c6b364829b33a4a9/brick
>Result: volume create: vol_4c71beb63ff6abb18f5ad259cbb0a53a: success: please start the volume to access data
>[negroni] Started GET /queue/065ad33878d141c01bb83ce000d7984d
>[negroni] Completed 200 OK in 134.025Âµs
>[negroni] Started GET /queue/065ad33878d141c01bb83ce000d7984d
>[negroni] Completed 200 OK in 134.287Âµs
>[negroni] Started GET /queue/065ad33878d141c01bb83ce000d7984d
>[negroni] Completed 200 OK in 133.463Âµs
>[negroni] Started GET /queue/065ad33878d141c01bb83ce000d7984d
>[negroni] Completed 200 OK in 133.276Âµs
>[kubeexec] DEBUG 2018/06/08 08:38:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: gluster --mode=script volume start vol_4c71beb63ff6abb18f5ad259cbb0a53a
>Result: volume start: vol_4c71beb63ff6abb18f5ad259cbb0a53a: success
>[heketi] INFO 2018/06/08 08:38:34 Create Volume succeeded
>[asynchttp] INFO 2018/06/08 08:38:34 asynchttp.go:292: Completed job 065ad33878d141c01bb83ce000d7984d in 5.623403765s
>[negroni] Started GET /queue/065ad33878d141c01bb83ce000d7984d
>[negroni] Completed 303 See Other in 217.954Âµs
>[negroni] Started GET /volumes/4c71beb63ff6abb18f5ad259cbb0a53a
>[negroni] Completed 200 OK in 5.362082ms
>[negroni] Started GET /clusters
>[negroni] Completed 200 OK in 185.627Âµs
>[negroni] Started GET /clusters/0a73c60efdd4673113b668afea101e6d
>[negroni] Completed 200 OK in 335.325Âµs
>[negroni] Started GET /nodes/278bd6b4e16a8e62ef15aaae22e6abc1
>[negroni] Completed 200 OK in 2.714256ms
>[negroni] Started GET /nodes/70423cc0bcd044fe5ba8bbbd256a3e49
>[negroni] Completed 200 OK in 1.592671ms
>[negroni] Started GET /nodes/d942fe6c0ee5691b7cc263968f97b650
>[negroni] Completed 200 OK in 894.631Âµs
>[negroni] Started GET /volumes/4c71beb63ff6abb18f5ad259cbb0a53a
>[negroni] Completed 200 OK in 733.605Âµs
>[negroni] Started POST /volumes/4c71beb63ff6abb18f5ad259cbb0a53a/expand
>[heketi] INFO 2018/06/08 08:38:36 Allocating brick set #0
>[negroni] Completed 202 Accepted in 12.804927ms
>[asynchttp] INFO 2018/06/08 08:38:36 asynchttp.go:288: Started job 680c1f99b6d8dd2df4c9a85721e53eb2
>[heketi] INFO 2018/06/08 08:38:36 Started async operation: Expand Volume
>[negroni] Started GET /queue/680c1f99b6d8dd2df4c9a85721e53eb2
>[negroni] Completed 200 OK in 105.034Âµs
>[heketi] INFO 2018/06/08 08:38:36 Creating brick f68fd071255bef02a0f31732a1eec3a3
>[heketi] INFO 2018/06/08 08:38:36 Creating brick 2ffaceaa46888863bcb20dfba9c6c722
>[heketi] INFO 2018/06/08 08:38:36 Creating brick 53721c3d4de0bf661138394e5374172c
>[kubeexec] DEBUG 2018/06/08 08:38:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir -p /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_2ffaceaa46888863bcb20dfba9c6c722
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:38:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir -p /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_53721c3d4de0bf661138394e5374172c
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:38:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir -p /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_f68fd071255bef02a0f31732a1eec3a3
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:38:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvcreate --autobackup=n --poolmetadatasize 28672K --chunksize 256K --size 5242880K --thin vg_b861af9cc67f5bf8d5a8f9e38f56b14e/tp_2ffaceaa46888863bcb20dfba9c6c722 --virtualsize 5242880K --name brick_2ffaceaa46888863bcb20dfba9c6c722
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_2ffaceaa46888863bcb20dfba9c6c722" created.
>[kubeexec] DEBUG 2018/06/08 08:38:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvcreate --autobackup=n --poolmetadatasize 28672K --chunksize 256K --size 5242880K --thin vg_66a9af9f6bac95f9d8d556a2f14c29d3/tp_f68fd071255bef02a0f31732a1eec3a3 --virtualsize 5242880K --name brick_f68fd071255bef02a0f31732a1eec3a3
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_f68fd071255bef02a0f31732a1eec3a3" created.
>[kubeexec] DEBUG 2018/06/08 08:38:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvcreate --autobackup=n --poolmetadatasize 28672K --chunksize 256K --size 5242880K --thin vg_96f1667f2f1ced2c5ef94772922be93b/tp_53721c3d4de0bf661138394e5374172c --virtualsize 5242880K --name brick_53721c3d4de0bf661138394e5374172c
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_53721c3d4de0bf661138394e5374172c" created.
>[kubeexec] DEBUG 2018/06/08 08:38:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_2ffaceaa46888863bcb20dfba9c6c722
>Result: meta-data=/dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_2ffaceaa46888863bcb20dfba9c6c722 isize=512    agcount=8, agsize=163840 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=1310720, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:38:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_53721c3d4de0bf661138394e5374172c
>Result: meta-data=/dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_53721c3d4de0bf661138394e5374172c isize=512    agcount=8, agsize=163840 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=1310720, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:38:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_f68fd071255bef02a0f31732a1eec3a3
>Result: meta-data=/dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_f68fd071255bef02a0f31732a1eec3a3 isize=512    agcount=8, agsize=163840 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=1310720, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:38:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: awk "BEGIN {print \"/dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_2ffaceaa46888863bcb20dfba9c6c722 /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_2ffaceaa46888863bcb20dfba9c6c722 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:38:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: awk "BEGIN {print \"/dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_53721c3d4de0bf661138394e5374172c /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_53721c3d4de0bf661138394e5374172c xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:38:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: awk "BEGIN {print \"/dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_f68fd071255bef02a0f31732a1eec3a3 /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_f68fd071255bef02a0f31732a1eec3a3 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:38:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_2ffaceaa46888863bcb20dfba9c6c722 /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_2ffaceaa46888863bcb20dfba9c6c722
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:38:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_53721c3d4de0bf661138394e5374172c /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_53721c3d4de0bf661138394e5374172c
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:38:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_f68fd071255bef02a0f31732a1eec3a3 /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_f68fd071255bef02a0f31732a1eec3a3
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:38:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_2ffaceaa46888863bcb20dfba9c6c722/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:38:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_53721c3d4de0bf661138394e5374172c/brick
>Result: 
>[negroni] Started GET /queue/680c1f99b6d8dd2df4c9a85721e53eb2
>[negroni] Completed 200 OK in 130.899Âµs
>[kubeexec] DEBUG 2018/06/08 08:38:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_f68fd071255bef02a0f31732a1eec3a3/brick
>Result: 
>[negroni] Started GET /queue/680c1f99b6d8dd2df4c9a85721e53eb2
>[negroni] Completed 200 OK in 201.445Âµs
>[negroni] Started GET /queue/680c1f99b6d8dd2df4c9a85721e53eb2
>[negroni] Completed 200 OK in 157.678Âµs
>[negroni] Started GET /queue/680c1f99b6d8dd2df4c9a85721e53eb2
>[negroni] Completed 200 OK in 150.198Âµs
>[negroni] Started GET /queue/680c1f99b6d8dd2df4c9a85721e53eb2
>[negroni] Completed 200 OK in 281.951Âµs
>[kubeexec] DEBUG 2018/06/08 08:38:42 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script volume add-brick vol_4c71beb63ff6abb18f5ad259cbb0a53a 10.70.46.122:/var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_f68fd071255bef02a0f31732a1eec3a3/brick 10.70.47.76:/var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_53721c3d4de0bf661138394e5374172c/brick 10.70.46.187:/var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_2ffaceaa46888863bcb20dfba9c6c722/brick
>Result: volume add-brick: success
>[negroni] Started GET /queue/680c1f99b6d8dd2df4c9a85721e53eb2
>[negroni] Completed 200 OK in 245.915Âµs
>[negroni] Started GET /queue/680c1f99b6d8dd2df4c9a85721e53eb2
>[negroni] Completed 200 OK in 131.308Âµs
>[negroni] Started GET /queue/680c1f99b6d8dd2df4c9a85721e53eb2
>[negroni] Completed 200 OK in 162.346Âµs
>[negroni] Started GET /queue/680c1f99b6d8dd2df4c9a85721e53eb2
>[negroni] Completed 200 OK in 212.625Âµs
>[negroni] Started GET /queue/680c1f99b6d8dd2df4c9a85721e53eb2
>[negroni] Completed 200 OK in 214.889Âµs
>[negroni] Started GET /queue/680c1f99b6d8dd2df4c9a85721e53eb2
>[negroni] Completed 200 OK in 226.111Âµs
>[negroni] Started GET /queue/680c1f99b6d8dd2df4c9a85721e53eb2
>[negroni] Completed 200 OK in 230.645Âµs
>[negroni] Started GET /queue/680c1f99b6d8dd2df4c9a85721e53eb2
>[negroni] Completed 200 OK in 178.388Âµs
>[negroni] Started GET /queue/680c1f99b6d8dd2df4c9a85721e53eb2
>[negroni] Completed 200 OK in 338.767Âµs
>[negroni] Started GET /queue/680c1f99b6d8dd2df4c9a85721e53eb2
>[negroni] Completed 200 OK in 221.181Âµs
>[kubeexec] DEBUG 2018/06/08 08:38:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script volume rebalance vol_4c71beb63ff6abb18f5ad259cbb0a53a start
>Result: volume rebalance: vol_4c71beb63ff6abb18f5ad259cbb0a53a: success: Rebalance on vol_4c71beb63ff6abb18f5ad259cbb0a53a has been started successfully. Use rebalance status command to check status of the rebalance process.
>ID: bd0a41ba-c7e1-4cf8-b736-e4424281746f
>
>[heketi] INFO 2018/06/08 08:38:52 Expand Volume succeeded
>[asynchttp] INFO 2018/06/08 08:38:52 asynchttp.go:292: Completed job 680c1f99b6d8dd2df4c9a85721e53eb2 in 15.690946098s
>[negroni] Started GET /queue/680c1f99b6d8dd2df4c9a85721e53eb2
>[negroni] Completed 303 See Other in 172.433Âµs
>[negroni] Started GET /volumes/4c71beb63ff6abb18f5ad259cbb0a53a
>[negroni] Completed 200 OK in 7.496832ms
>[negroni] Started GET /clusters
>[negroni] Completed 200 OK in 243.818Âµs
>[negroni] Started GET /clusters/0a73c60efdd4673113b668afea101e6d
>[negroni] Completed 200 OK in 385.039Âµs
>[negroni] Started GET /nodes/278bd6b4e16a8e62ef15aaae22e6abc1
>[negroni] Completed 200 OK in 3.990933ms
>[negroni] Started GET /nodes/70423cc0bcd044fe5ba8bbbd256a3e49
>[negroni] Completed 200 OK in 2.636697ms
>[negroni] Started GET /nodes/d942fe6c0ee5691b7cc263968f97b650
>[negroni] Completed 200 OK in 1.671794ms
>[negroni] Started GET /volumes/4c71beb63ff6abb18f5ad259cbb0a53a
>[negroni] Completed 200 OK in 1.625615ms
>[negroni] Started DELETE /volumes/4c71beb63ff6abb18f5ad259cbb0a53a
>[negroni] Completed 202 Accepted in 12.042941ms
>[asynchttp] INFO 2018/06/08 08:38:56 asynchttp.go:288: Started job 4c80322fad4a5f64b73a0caab03c74c2
>[heketi] INFO 2018/06/08 08:38:56 Started async operation: Delete Volume
>[negroni] Started GET /queue/4c80322fad4a5f64b73a0caab03c74c2
>[negroni] Completed 200 OK in 114.618Âµs
>[kubeexec] DEBUG 2018/06/08 08:38:56 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: gluster --mode=script snapshot list vol_4c71beb63ff6abb18f5ad259cbb0a53a --xml
>Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
><cliOutput>
>  <opRet>0</opRet>
>  <opErrno>0</opErrno>
>  <opErrstr/>
>  <snapList>
>    <count>0</count>
>  </snapList>
></cliOutput>
>[negroni] Started GET /queue/4c80322fad4a5f64b73a0caab03c74c2
>[negroni] Completed 200 OK in 215.402Âµs
>[negroni] Started GET /queue/4c80322fad4a5f64b73a0caab03c74c2
>[negroni] Completed 200 OK in 216.558Âµs
>[kubeexec] DEBUG 2018/06/08 08:38:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: gluster --mode=script volume stop vol_4c71beb63ff6abb18f5ad259cbb0a53a force
>Result: volume stop: vol_4c71beb63ff6abb18f5ad259cbb0a53a: success
>[kubeexec] DEBUG 2018/06/08 08:38:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: gluster --mode=script volume delete vol_4c71beb63ff6abb18f5ad259cbb0a53a
>Result: volume delete: vol_4c71beb63ff6abb18f5ad259cbb0a53a: success
>[heketi] INFO 2018/06/08 08:38:59 Deleting brick de28e8f1db02df454626ab7304cd17b6
>[heketi] INFO 2018/06/08 08:38:59 Deleting brick 2ffaceaa46888863bcb20dfba9c6c722
>[heketi] INFO 2018/06/08 08:38:59 Deleting brick 53721c3d4de0bf661138394e5374172c
>[heketi] INFO 2018/06/08 08:38:59 Deleting brick e106612b2d23de31c6b364829b33a4a9
>[heketi] INFO 2018/06/08 08:38:59 Deleting brick f68fd071255bef02a0f31732a1eec3a3
>[heketi] INFO 2018/06/08 08:38:59 Deleting brick 88c50c0f97424a3116d7ecf9870b0f3e
>[kubeexec] DEBUG 2018/06/08 08:38:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mount | grep -w /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_de28e8f1db02df454626ab7304cd17b6 | cut -d" " -f1
>Result: /dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_de28e8f1db02df454626ab7304cd17b6
>[kubeexec] DEBUG 2018/06/08 08:38:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mount | grep -w /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_53721c3d4de0bf661138394e5374172c | cut -d" " -f1
>Result: /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_53721c3d4de0bf661138394e5374172c
>[kubeexec] DEBUG 2018/06/08 08:38:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mount | grep -w /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_2ffaceaa46888863bcb20dfba9c6c722 | cut -d" " -f1
>Result: /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_2ffaceaa46888863bcb20dfba9c6c722
>[negroni] Started GET /queue/4c80322fad4a5f64b73a0caab03c74c2
>[negroni] Completed 200 OK in 221.045Âµs
>[kubeexec] DEBUG 2018/06/08 08:38:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mount | grep -w /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_e106612b2d23de31c6b364829b33a4a9 | cut -d" " -f1
>Result: /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_e106612b2d23de31c6b364829b33a4a9
>[kubeexec] DEBUG 2018/06/08 08:38:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mount | grep -w /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_f68fd071255bef02a0f31732a1eec3a3 | cut -d" " -f1
>Result: /dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_f68fd071255bef02a0f31732a1eec3a3
>[kubeexec] DEBUG 2018/06/08 08:39:00 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mount | grep -w /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_88c50c0f97424a3116d7ecf9870b0f3e | cut -d" " -f1
>Result: /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_88c50c0f97424a3116d7ecf9870b0f3e
>[kubeexec] DEBUG 2018/06/08 08:39:00 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_53721c3d4de0bf661138394e5374172c
>
>Result:   vg_96f1667f2f1ced2c5ef94772922be93b/tp_53721c3d4de0bf661138394e5374172c
>[negroni] Started GET /queue/4c80322fad4a5f64b73a0caab03c74c2
>[negroni] Completed 200 OK in 135.089Âµs
>[kubeexec] DEBUG 2018/06/08 08:39:00 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_de28e8f1db02df454626ab7304cd17b6
>
>Result:   vg_66a9af9f6bac95f9d8d556a2f14c29d3/tp_de28e8f1db02df454626ab7304cd17b6
>[kubeexec] DEBUG 2018/06/08 08:39:01 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_2ffaceaa46888863bcb20dfba9c6c722
>
>Result:   vg_b861af9cc67f5bf8d5a8f9e38f56b14e/tp_2ffaceaa46888863bcb20dfba9c6c722
>[negroni] Started GET /queue/4c80322fad4a5f64b73a0caab03c74c2
>[negroni] Completed 200 OK in 131.266Âµs
>[kubeexec] DEBUG 2018/06/08 08:39:01 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_e106612b2d23de31c6b364829b33a4a9
>
>Result:   vg_96f1667f2f1ced2c5ef94772922be93b/tp_e106612b2d23de31c6b364829b33a4a9
>[kubeexec] DEBUG 2018/06/08 08:39:02 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_f68fd071255bef02a0f31732a1eec3a3
>
>Result:   vg_66a9af9f6bac95f9d8d556a2f14c29d3/tp_f68fd071255bef02a0f31732a1eec3a3
>[negroni] Started GET /queue/4c80322fad4a5f64b73a0caab03c74c2
>[negroni] Completed 200 OK in 356.061Âµs
>[kubeexec] DEBUG 2018/06/08 08:39:02 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_88c50c0f97424a3116d7ecf9870b0f3e
>
>Result:   vg_d389f0278a774bd7443a09af960961d8/tp_88c50c0f97424a3116d7ecf9870b0f3e
>[kubeexec] DEBUG 2018/06/08 08:39:03 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: umount /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_53721c3d4de0bf661138394e5374172c
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:39:03 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: umount /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_de28e8f1db02df454626ab7304cd17b6
>Result: 
>[negroni] Started GET /queue/4c80322fad4a5f64b73a0caab03c74c2
>[negroni] Completed 200 OK in 177.645Âµs
>[kubeexec] DEBUG 2018/06/08 08:39:04 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: umount /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_2ffaceaa46888863bcb20dfba9c6c722
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:39:04 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: umount /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_e106612b2d23de31c6b364829b33a4a9
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:39:04 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: umount /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_f68fd071255bef02a0f31732a1eec3a3
>Result: 
>[negroni] Started GET /queue/4c80322fad4a5f64b73a0caab03c74c2
>[negroni] Completed 200 OK in 198.875Âµs
>[kubeexec] DEBUG 2018/06/08 08:39:05 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: umount /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_88c50c0f97424a3116d7ecf9870b0f3e
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:39:05 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: sed -i.save "/brick_53721c3d4de0bf661138394e5374172c/d" /var/lib/heketi/fstab
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:39:05 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: sed -i.save "/brick_de28e8f1db02df454626ab7304cd17b6/d" /var/lib/heketi/fstab
>Result: 
>[negroni] Started GET /queue/4c80322fad4a5f64b73a0caab03c74c2
>[negroni] Completed 200 OK in 142.025Âµs
>[kubeexec] DEBUG 2018/06/08 08:39:06 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: sed -i.save "/brick_2ffaceaa46888863bcb20dfba9c6c722/d" /var/lib/heketi/fstab
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:39:06 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: sed -i.save "/brick_e106612b2d23de31c6b364829b33a4a9/d" /var/lib/heketi/fstab
>Result: 
>[negroni] Started GET /queue/4c80322fad4a5f64b73a0caab03c74c2
>[negroni] Completed 200 OK in 202.349Âµs
>[kubeexec] DEBUG 2018/06/08 08:39:06 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: sed -i.save "/brick_f68fd071255bef02a0f31732a1eec3a3/d" /var/lib/heketi/fstab
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:39:07 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: sed -i.save "/brick_88c50c0f97424a3116d7ecf9870b0f3e/d" /var/lib/heketi/fstab
>Result: 
>[negroni] Started GET /queue/4c80322fad4a5f64b73a0caab03c74c2
>[negroni] Completed 200 OK in 219.098Âµs
>[kubeexec] DEBUG 2018/06/08 08:39:07 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvremove --autobackup=n -f /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_53721c3d4de0bf661138394e5374172c
>
>Result:   Logical volume "brick_53721c3d4de0bf661138394e5374172c" successfully removed
>[kubeexec] DEBUG 2018/06/08 08:39:08 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvremove --autobackup=n -f /dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_de28e8f1db02df454626ab7304cd17b6
>
>Result:   Logical volume "brick_de28e8f1db02df454626ab7304cd17b6" successfully removed
>[negroni] Started GET /queue/4c80322fad4a5f64b73a0caab03c74c2
>[negroni] Completed 200 OK in 160.855Âµs
>[kubeexec] DEBUG 2018/06/08 08:39:08 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvremove --autobackup=n -f /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_2ffaceaa46888863bcb20dfba9c6c722
>
>Result:   Logical volume "brick_2ffaceaa46888863bcb20dfba9c6c722" successfully removed
>[kubeexec] DEBUG 2018/06/08 08:39:09 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvremove --autobackup=n -f /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_e106612b2d23de31c6b364829b33a4a9
>
>Result:   Logical volume "brick_e106612b2d23de31c6b364829b33a4a9" successfully removed
>[kubeexec] DEBUG 2018/06/08 08:39:09 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvremove --autobackup=n -f /dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_f68fd071255bef02a0f31732a1eec3a3
>
>Result:   Logical volume "brick_f68fd071255bef02a0f31732a1eec3a3" successfully removed
>[negroni] Started GET /queue/4c80322fad4a5f64b73a0caab03c74c2
>[negroni] Completed 200 OK in 266.45Âµs
>[kubeexec] DEBUG 2018/06/08 08:39:10 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvremove --autobackup=n -f /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_88c50c0f97424a3116d7ecf9870b0f3e
>
>Result:   Logical volume "brick_88c50c0f97424a3116d7ecf9870b0f3e" successfully removed
>[kubeexec] DEBUG 2018/06/08 08:39:10 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvs --noheadings --options=thin_count   vg_96f1667f2f1ced2c5ef94772922be93b/tp_53721c3d4de0bf661138394e5374172c
>
>Result:        0
>[kubeexec] DEBUG 2018/06/08 08:39:10 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvs --noheadings --options=thin_count   vg_66a9af9f6bac95f9d8d556a2f14c29d3/tp_de28e8f1db02df454626ab7304cd17b6
>
>Result:        0
>[negroni] Started GET /queue/4c80322fad4a5f64b73a0caab03c74c2
>[negroni] Completed 200 OK in 195.7Âµs
>[kubeexec] DEBUG 2018/06/08 08:39:11 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvs --noheadings --options=thin_count   vg_b861af9cc67f5bf8d5a8f9e38f56b14e/tp_2ffaceaa46888863bcb20dfba9c6c722
>
>Result:        0
>[kubeexec] DEBUG 2018/06/08 08:39:11 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvs --noheadings --options=thin_count   vg_96f1667f2f1ced2c5ef94772922be93b/tp_e106612b2d23de31c6b364829b33a4a9
>
>Result:        0
>[kubeexec] DEBUG 2018/06/08 08:39:11 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvs --noheadings --options=thin_count   vg_66a9af9f6bac95f9d8d556a2f14c29d3/tp_f68fd071255bef02a0f31732a1eec3a3
>
>Result:        0
>[negroni] Started GET /queue/4c80322fad4a5f64b73a0caab03c74c2
>[negroni] Completed 200 OK in 156.714Âµs
>[kubeexec] DEBUG 2018/06/08 08:39:12 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvs --noheadings --options=thin_count   vg_d389f0278a774bd7443a09af960961d8/tp_88c50c0f97424a3116d7ecf9870b0f3e
>
>Result:        0
>[negroni] Started GET /queue/4c80322fad4a5f64b73a0caab03c74c2
>[negroni] Completed 200 OK in 146.359Âµs
>[kubeexec] DEBUG 2018/06/08 08:39:12 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvremove --autobackup=n -f   vg_96f1667f2f1ced2c5ef94772922be93b/tp_53721c3d4de0bf661138394e5374172c
>
>Result:   Logical volume "tp_53721c3d4de0bf661138394e5374172c" successfully removed
>[kubeexec] DEBUG 2018/06/08 08:39:12 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvremove --autobackup=n -f   vg_66a9af9f6bac95f9d8d556a2f14c29d3/tp_de28e8f1db02df454626ab7304cd17b6
>
>Result:   Logical volume "tp_de28e8f1db02df454626ab7304cd17b6" successfully removed
>[kubeexec] DEBUG 2018/06/08 08:39:13 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvremove --autobackup=n -f   vg_b861af9cc67f5bf8d5a8f9e38f56b14e/tp_2ffaceaa46888863bcb20dfba9c6c722
>
>Result:   Logical volume "tp_2ffaceaa46888863bcb20dfba9c6c722" successfully removed
>[negroni] Started GET /queue/4c80322fad4a5f64b73a0caab03c74c2
>[negroni] Completed 200 OK in 164.818Âµs
>[kubeexec] DEBUG 2018/06/08 08:39:13 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvremove --autobackup=n -f   vg_96f1667f2f1ced2c5ef94772922be93b/tp_e106612b2d23de31c6b364829b33a4a9
>
>Result:   Logical volume "tp_e106612b2d23de31c6b364829b33a4a9" successfully removed
>[kubeexec] DEBUG 2018/06/08 08:39:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvremove --autobackup=n -f   vg_66a9af9f6bac95f9d8d556a2f14c29d3/tp_f68fd071255bef02a0f31732a1eec3a3
>
>Result:   Logical volume "tp_f68fd071255bef02a0f31732a1eec3a3" successfully removed
>[heketi] INFO 2018/06/08 08:39:14 Starting Node Health Status refresh
>[cmdexec] INFO 2018/06/08 08:39:14 Check Glusterd service status in node dhcp46-187.lab.eng.blr.redhat.com
>[negroni] Started GET /queue/4c80322fad4a5f64b73a0caab03c74c2
>[negroni] Completed 200 OK in 220.665Âµs
>[kubeexec] DEBUG 2018/06/08 08:39:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvremove --autobackup=n -f   vg_d389f0278a774bd7443a09af960961d8/tp_88c50c0f97424a3116d7ecf9870b0f3e
>
>Result:   Logical volume "tp_88c50c0f97424a3116d7ecf9870b0f3e" successfully removed
>[kubeexec] DEBUG 2018/06/08 08:39:15 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: rmdir /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_53721c3d4de0bf661138394e5374172c
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:39:15 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: rmdir /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_de28e8f1db02df454626ab7304cd17b6
>Result: 
>[negroni] Started GET /queue/4c80322fad4a5f64b73a0caab03c74c2
>[negroni] Completed 200 OK in 215.198Âµs
>[kubeexec] DEBUG 2018/06/08 08:39:15 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: rmdir /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_2ffaceaa46888863bcb20dfba9c6c722
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:39:16 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: rmdir /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_e106612b2d23de31c6b364829b33a4a9
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:39:16 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: rmdir /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_f68fd071255bef02a0f31732a1eec3a3
>Result: 
>[negroni] Started GET /queue/4c80322fad4a5f64b73a0caab03c74c2
>[negroni] Completed 200 OK in 214.323Âµs
>[kubeexec] DEBUG 2018/06/08 08:39:16 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:33 UTC; 2h 15min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48fed567_6ae4_11e8_bc4b_005056a5f18a.slice/docker-a506d9aaca4b440d184df40d01e90b2040811c8090bf4c19d892081307b28c0c.scope/system.slice/glusterd.service
>           ââ  433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1228 /usr/sbin/glusterfsd -s 10.70.46.187 --volfile-id heketidbstorage.10.70.46.187.var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.187-var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.pid -S /var/run/gluster/c08898c2e3b0190c62df3a9d4a013c92.socket --brick-name /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.log --xlator-option *-posix.glusterd-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ11892 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/8fb729a0bbd51b204491c603c84acaa4.socket --xlator-option *replicate*.node-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034
>
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Unit glusterd.service entered failed state.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: glusterd.service failed.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:33 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 08:39:16 Periodic health check status: node 278bd6b4e16a8e62ef15aaae22e6abc1 up=true
>[cmdexec] INFO 2018/06/08 08:39:16 Check Glusterd service status in node dhcp46-122.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 08:39:17 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:42 UTC; 2h 15min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4900c968_6ae4_11e8_bc4b_005056a5f18a.slice/docker-438fb5fb16a82a3e701b28fa1f1ef8623dc37ca06374c604e5781a2b2ab2e06d.scope/system.slice/glusterd.service
>           ââ  433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1106 /usr/sbin/glusterfsd -s 10.70.46.122 --volfile-id heketidbstorage.10.70.46.122.var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.122-var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.pid -S /var/run/gluster/9fa3ac3b12826912287adc88c26ac134.socket --brick-name /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_29e5b36d921cca1b0699871d9727c932/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.log --xlator-option *-posix.glusterd-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ12039 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f8994cdb09351d83e70da08261d99c64.socket --xlator-option *replicate*.node-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0
>
>Jun 08 06:23:40 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:42 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 08:39:17 Periodic health check status: node 70423cc0bcd044fe5ba8bbbd256a3e49 up=true
>[cmdexec] INFO 2018/06/08 08:39:17 Check Glusterd service status in node dhcp47-76.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 08:39:17 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: rmdir /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_88c50c0f97424a3116d7ecf9870b0f3e
>Result: 
>[heketi] INFO 2018/06/08 08:39:17 Delete Volume succeeded
>[asynchttp] INFO 2018/06/08 08:39:17 asynchttp.go:292: Completed job 4c80322fad4a5f64b73a0caab03c74c2 in 21.051068141s
>[negroni] Started GET /queue/4c80322fad4a5f64b73a0caab03c74c2
>[negroni] Completed 204 No Content in 242.134Âµs
>[negroni] Started GET /clusters
>[negroni] Completed 200 OK in 2.040568ms
>[negroni] Started GET /clusters/0a73c60efdd4673113b668afea101e6d
>[negroni] Completed 200 OK in 482.276Âµs
>[negroni] Started GET /nodes/278bd6b4e16a8e62ef15aaae22e6abc1
>[negroni] Completed 200 OK in 6.934401ms
>[kubeexec] DEBUG 2018/06/08 08:39:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 07:25:20 UTC; 1h 13min ago
>  Process: 828 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 830 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5cb2299_6aec_11e8_bc4b_005056a5f18a.slice/docker-7f9223abc9a8641cea68f374f4039235515978ee2dab970ac6cbef182c535426.scope/system.slice/glusterd.service
>           ââ  830 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1197 /usr/sbin/glusterfsd -s 10.70.47.76 --volfile-id heketidbstorage.10.70.47.76.var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.76-var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.pid -S /var/run/gluster/b6976f9ddc3abc90e1dfa91c35a7c611.socket --brick-name /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_54c06cdf52c6fe1e3a3f8a9d39cde33d/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.log --xlator-option *-posix.glusterd-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ10345 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/ec468453b11caed5259f5e1abcc65b44.socket --xlator-option *replicate*.node-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94
>
>Jun 08 07:25:17 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 07:25:20 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 08:39:18 Periodic health check status: node d942fe6c0ee5691b7cc263968f97b650 up=true
>[heketi] INFO 2018/06/08 08:39:18 Cleaned 0 nodes from health cache
>[negroni] Started GET /nodes/70423cc0bcd044fe5ba8bbbd256a3e49
>[negroni] Completed 200 OK in 2.388622ms
>[negroni] Started GET /nodes/d942fe6c0ee5691b7cc263968f97b650
>[negroni] Completed 200 OK in 3.121456ms
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 908.335Âµs
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 1.136091ms
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 710.48Âµs
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 1.258828ms
>[negroni] Started POST /volumes
>[negroni] Completed 401 Unauthorized in 150.882Âµs
>[negroni] Started POST /volumes
>[negroni] Completed 401 Unauthorized in 124.466Âµs
>[negroni] Started POST /volumes
>[negroni] Completed 401 Unauthorized in 127.266Âµs
>[negroni] Started POST /volumes
>[negroni] Completed 401 Unauthorized in 87.133Âµs
>[negroni] Started POST /volumes
>[negroni] Completed 401 Unauthorized in 85.074Âµs
>[negroni] Started POST /volumes
>[negroni] Completed 401 Unauthorized in 79.618Âµs
>[negroni] Started POST /volumes
>[negroni] Completed 401 Unauthorized in 54.664Âµs
>[negroni] Started POST /volumes
>[negroni] Completed 401 Unauthorized in 126.224Âµs
>[negroni] Started POST /volumes
>[negroni] Completed 401 Unauthorized in 99.066Âµs
>[negroni] Started POST /volumes
>[negroni] Completed 401 Unauthorized in 121.055Âµs
>[negroni] Started POST /volumes
>[negroni] Completed 401 Unauthorized in 94.311Âµs
>[negroni] Started POST /volumes
>[negroni] Completed 401 Unauthorized in 115.706Âµs
>[negroni] Started POST /volumes
>[negroni] Completed 401 Unauthorized in 132.679Âµs
>[negroni] Started POST /volumes
>[negroni] Completed 401 Unauthorized in 67.59Âµs
>[negroni] Started POST /volumes
>[negroni] Completed 401 Unauthorized in 47.642Âµs
>[negroni] Started POST /volumes
>[negroni] Completed 401 Unauthorized in 49.557Âµs
>[negroni] Started POST /volumes
>[negroni] Completed 401 Unauthorized in 136.136Âµs
>[negroni] Started POST /volumes
>[negroni] Completed 401 Unauthorized in 142.9Âµs
>[negroni] Started POST /volumes
>[negroni] Completed 401 Unauthorized in 106.699Âµs
>[negroni] Started POST /volumes
>[negroni] Completed 401 Unauthorized in 100.489Âµs
>[negroni] Started POST /volumes
>[negroni] Completed 401 Unauthorized in 137.379Âµs
>[negroni] Started POST /volumes
>[negroni] Completed 401 Unauthorized in 130.053Âµs
>[negroni] Started POST /volumes
>[negroni] Completed 401 Unauthorized in 146.979Âµs
>[negroni] Started POST /volumes
>[negroni] Completed 401 Unauthorized in 191.131Âµs
>[negroni] Started POST /volumes
>[negroni] Completed 401 Unauthorized in 90.736Âµs
>[negroni] Started POST /volumes
>[negroni] Completed 401 Unauthorized in 68.559Âµs
>[negroni] Started POST /volumes
>[negroni] Completed 401 Unauthorized in 54.344Âµs
>[negroni] Started POST /volumes
>[negroni] Completed 401 Unauthorized in 76.897Âµs
>[negroni] Started POST /volumes
>[negroni] Started POST /volumes
>[negroni] Completed 401 Unauthorized in 98.85Âµs
>[negroni] Completed 401 Unauthorized in 51.989Âµs
>[negroni] Started POST /volumes
>[negroni] Completed 401 Unauthorized in 71.79Âµs
>[negroni] Started POST /volumes
>[negroni] Completed 401 Unauthorized in 84.853Âµs
>[negroni] Started POST /volumes
>[negroni] Started POST /volumes
>[negroni] Completed 401 Unauthorized in 103.171Âµs
>[negroni] Completed 401 Unauthorized in 67.285Âµs
>[negroni] Started POST /volumes
>[negroni] Completed 401 Unauthorized in 178.836Âµs
>[negroni] Started POST /volumes
>[negroni] Completed 401 Unauthorized in 148.089Âµs
>[negroni] Started POST /volumes
>[negroni] Completed 401 Unauthorized in 81.364Âµs
>[negroni] Started POST /volumes
>[negroni] Completed 401 Unauthorized in 46.169Âµs
>[negroni] Started POST /volumes
>[negroni] Completed 401 Unauthorized in 58.222Âµs
>[negroni] Started POST /volumes
>[negroni] Completed 401 Unauthorized in 43.287Âµs
>[negroni] Started POST /volumes
>[negroni] Completed 401 Unauthorized in 127.246Âµs
>[negroni] Started POST /volumes
>[negroni] Completed 401 Unauthorized in 66.499Âµs
>[negroni] Started POST /volumes
>[negroni] Completed 401 Unauthorized in 90.786Âµs
>[negroni] Started POST /volumes
>[negroni] Completed 401 Unauthorized in 68.65Âµs
>[negroni] Started POST /volumes
>[negroni] Completed 401 Unauthorized in 160.454Âµs
>[negroni] Started POST /volumes
>[negroni] Completed 401 Unauthorized in 132.312Âµs
>[negroni] Started POST /volumes
>[negroni] Started POST /volumes
>[negroni] Completed 401 Unauthorized in 142.997Âµs
>[negroni] Completed 401 Unauthorized in 93.143Âµs
>[negroni] Started POST /volumes
>[negroni] Completed 401 Unauthorized in 98.415Âµs
>[negroni] Started POST /volumes
>[negroni] Completed 401 Unauthorized in 91.53Âµs
>[negroni] Started POST /volumes
>[negroni] Completed 401 Unauthorized in 100.242Âµs
>[negroni] Started POST /volumes
>[negroni] Completed 401 Unauthorized in 88.342Âµs
>[heketi] INFO 2018/06/08 08:41:14 Starting Node Health Status refresh
>[cmdexec] INFO 2018/06/08 08:41:14 Check Glusterd service status in node dhcp46-187.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 08:41:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:33 UTC; 2h 17min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48fed567_6ae4_11e8_bc4b_005056a5f18a.slice/docker-a506d9aaca4b440d184df40d01e90b2040811c8090bf4c19d892081307b28c0c.scope/system.slice/glusterd.service
>           ââ  433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1228 /usr/sbin/glusterfsd -s 10.70.46.187 --volfile-id heketidbstorage.10.70.46.187.var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.187-var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.pid -S /var/run/gluster/c08898c2e3b0190c62df3a9d4a013c92.socket --brick-name /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.log --xlator-option *-posix.glusterd-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ11892 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/8fb729a0bbd51b204491c603c84acaa4.socket --xlator-option *replicate*.node-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034
>
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Unit glusterd.service entered failed state.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: glusterd.service failed.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:33 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 08:41:14 Periodic health check status: node 278bd6b4e16a8e62ef15aaae22e6abc1 up=true
>[cmdexec] INFO 2018/06/08 08:41:14 Check Glusterd service status in node dhcp46-122.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 08:41:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:42 UTC; 2h 17min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4900c968_6ae4_11e8_bc4b_005056a5f18a.slice/docker-438fb5fb16a82a3e701b28fa1f1ef8623dc37ca06374c604e5781a2b2ab2e06d.scope/system.slice/glusterd.service
>           ââ  433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1106 /usr/sbin/glusterfsd -s 10.70.46.122 --volfile-id heketidbstorage.10.70.46.122.var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.122-var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.pid -S /var/run/gluster/9fa3ac3b12826912287adc88c26ac134.socket --brick-name /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_29e5b36d921cca1b0699871d9727c932/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.log --xlator-option *-posix.glusterd-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ12039 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f8994cdb09351d83e70da08261d99c64.socket --xlator-option *replicate*.node-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0
>
>Jun 08 06:23:40 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:42 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 08:41:14 Periodic health check status: node 70423cc0bcd044fe5ba8bbbd256a3e49 up=true
>[cmdexec] INFO 2018/06/08 08:41:14 Check Glusterd service status in node dhcp47-76.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 08:41:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 07:25:20 UTC; 1h 15min ago
>  Process: 828 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 830 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5cb2299_6aec_11e8_bc4b_005056a5f18a.slice/docker-7f9223abc9a8641cea68f374f4039235515978ee2dab970ac6cbef182c535426.scope/system.slice/glusterd.service
>           ââ  830 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1197 /usr/sbin/glusterfsd -s 10.70.47.76 --volfile-id heketidbstorage.10.70.47.76.var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.76-var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.pid -S /var/run/gluster/b6976f9ddc3abc90e1dfa91c35a7c611.socket --brick-name /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_54c06cdf52c6fe1e3a3f8a9d39cde33d/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.log --xlator-option *-posix.glusterd-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ10345 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/ec468453b11caed5259f5e1abcc65b44.socket --xlator-option *replicate*.node-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94
>
>Jun 08 07:25:17 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 07:25:20 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 08:41:14 Periodic health check status: node d942fe6c0ee5691b7cc263968f97b650 up=true
>[heketi] INFO 2018/06/08 08:41:14 Cleaned 0 nodes from health cache
>[negroni] Started POST /volumes
>[negroni] Started POST /volumes
>[negroni] Completed 401 Unauthorized in 180.426Âµs
>[negroni] Completed 401 Unauthorized in 116.966Âµs
>[negroni] Started POST /volumes
>[negroni] Completed 401 Unauthorized in 122.222Âµs
>[negroni] Started POST /volumes
>[negroni] Started POST /volumes
>[negroni] Completed 401 Unauthorized in 99.31Âµs
>[negroni] Completed 401 Unauthorized in 96.233Âµs
>[negroni] Started POST /volumes
>[negroni] Completed 401 Unauthorized in 99.012Âµs
>[negroni] Started POST /volumes
>[negroni] Completed 401 Unauthorized in 123.42Âµs
>[negroni] Started POST /volumes
>[negroni] Completed 401 Unauthorized in 114.021Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 165.274Âµs
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 698.161Âµs
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 1.024369ms
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 610.34Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 182.045Âµs
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 746.191Âµs
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 748.226Âµs
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 604.155Âµs
>[negroni] Started POST /volumes
>[heketi] INFO 2018/06/08 08:41:33 Allocating brick set #0
>[negroni] Completed 202 Accepted in 12.884009ms
>[asynchttp] INFO 2018/06/08 08:41:33 asynchttp.go:288: Started job e762df464aa012622774563dece1d0c2
>[heketi] INFO 2018/06/08 08:41:33 Started async operation: Create Volume
>[negroni] Started GET /queue/e762df464aa012622774563dece1d0c2
>[negroni] Completed 200 OK in 147.179Âµs
>[heketi] INFO 2018/06/08 08:41:33 Creating brick 4cb2a20a79c4ba5f51a7af73c72792d8
>[heketi] INFO 2018/06/08 08:41:33 Creating brick d093b076a37d3e3ba22944bd34ed0f9e
>[heketi] INFO 2018/06/08 08:41:33 Creating brick 487a6210cd6beafaea7d458f0edc796f
>[kubeexec] DEBUG 2018/06/08 08:41:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir -p /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_487a6210cd6beafaea7d458f0edc796f
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:41:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir -p /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_d093b076a37d3e3ba22944bd34ed0f9e
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:41:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir -p /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_4cb2a20a79c4ba5f51a7af73c72792d8
>Result: 
>[negroni] Started POST /volumes
>[heketi] INFO 2018/06/08 08:41:33 Allocating brick set #0
>[negroni] Completed 202 Accepted in 33.142296ms
>[asynchttp] INFO 2018/06/08 08:41:33 asynchttp.go:288: Started job e6d8d4b63c9bfa929b21498f22af956e
>[heketi] INFO 2018/06/08 08:41:33 Started async operation: Create Volume
>[negroni] Started GET /queue/e6d8d4b63c9bfa929b21498f22af956e
>[negroni] Completed 200 OK in 107.692Âµs
>[heketi] INFO 2018/06/08 08:41:33 Creating brick ff2a5406a86cd765269a1b519b76a810
>[heketi] INFO 2018/06/08 08:41:33 Creating brick ac730765fa6d170eb90c9795eaf5a9c5
>[heketi] INFO 2018/06/08 08:41:33 Creating brick 300c73448ebb5017a216c6661178ca8c
>[kubeexec] DEBUG 2018/06/08 08:41:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_3a4297677881963e3f80124971d50eea/tp_487a6210cd6beafaea7d458f0edc796f --virtualsize 2097152K --name brick_487a6210cd6beafaea7d458f0edc796f
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_487a6210cd6beafaea7d458f0edc796f" created.
>[kubeexec] DEBUG 2018/06/08 08:41:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_b861af9cc67f5bf8d5a8f9e38f56b14e/tp_d093b076a37d3e3ba22944bd34ed0f9e --virtualsize 2097152K --name brick_d093b076a37d3e3ba22944bd34ed0f9e
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_d093b076a37d3e3ba22944bd34ed0f9e" created.
>[kubeexec] DEBUG 2018/06/08 08:41:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_96f1667f2f1ced2c5ef94772922be93b/tp_4cb2a20a79c4ba5f51a7af73c72792d8 --virtualsize 2097152K --name brick_4cb2a20a79c4ba5f51a7af73c72792d8
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_4cb2a20a79c4ba5f51a7af73c72792d8" created.
>[kubeexec] DEBUG 2018/06/08 08:41:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_487a6210cd6beafaea7d458f0edc796f
>Result: meta-data=/dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_487a6210cd6beafaea7d458f0edc796f isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:41:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_d093b076a37d3e3ba22944bd34ed0f9e
>Result: meta-data=/dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_d093b076a37d3e3ba22944bd34ed0f9e isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:41:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_4cb2a20a79c4ba5f51a7af73c72792d8
>Result: meta-data=/dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_4cb2a20a79c4ba5f51a7af73c72792d8 isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:41:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: awk "BEGIN {print \"/dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_487a6210cd6beafaea7d458f0edc796f /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_487a6210cd6beafaea7d458f0edc796f xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:41:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: awk "BEGIN {print \"/dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_d093b076a37d3e3ba22944bd34ed0f9e /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_d093b076a37d3e3ba22944bd34ed0f9e xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:41:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: awk "BEGIN {print \"/dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_4cb2a20a79c4ba5f51a7af73c72792d8 /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_4cb2a20a79c4ba5f51a7af73c72792d8 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:41:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_d093b076a37d3e3ba22944bd34ed0f9e /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_d093b076a37d3e3ba22944bd34ed0f9e
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:41:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_487a6210cd6beafaea7d458f0edc796f /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_487a6210cd6beafaea7d458f0edc796f
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:41:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_4cb2a20a79c4ba5f51a7af73c72792d8 /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_4cb2a20a79c4ba5f51a7af73c72792d8
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:41:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_487a6210cd6beafaea7d458f0edc796f/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:41:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_d093b076a37d3e3ba22944bd34ed0f9e/brick
>Result: 
>[negroni] Started GET /queue/e762df464aa012622774563dece1d0c2
>[negroni] Completed 200 OK in 102.786Âµs
>[kubeexec] DEBUG 2018/06/08 08:41:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_4cb2a20a79c4ba5f51a7af73c72792d8/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:41:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: chown :2000 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_487a6210cd6beafaea7d458f0edc796f/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:41:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: chown :2000 /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_d093b076a37d3e3ba22944bd34ed0f9e/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:41:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: chown :2000 /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_4cb2a20a79c4ba5f51a7af73c72792d8/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:41:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: chmod 2775 /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_d093b076a37d3e3ba22944bd34ed0f9e/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:41:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: chmod 2775 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_487a6210cd6beafaea7d458f0edc796f/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:41:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: chmod 2775 /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_4cb2a20a79c4ba5f51a7af73c72792d8/brick
>Result: 
>[cmdexec] INFO 2018/06/08 08:41:34 Creating volume vol_b1176cd4e68f9cd8d5ed06cbfd1d12b5 replica 3
>[kubeexec] DEBUG 2018/06/08 08:41:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir -p /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_ff2a5406a86cd765269a1b519b76a810
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:41:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir -p /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_ac730765fa6d170eb90c9795eaf5a9c5
>Result: 
>[negroni] Started GET /queue/e6d8d4b63c9bfa929b21498f22af956e
>[negroni] Completed 200 OK in 104.554Âµs
>[kubeexec] DEBUG 2018/06/08 08:41:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir -p /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_300c73448ebb5017a216c6661178ca8c
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:41:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_66a9af9f6bac95f9d8d556a2f14c29d3/tp_ac730765fa6d170eb90c9795eaf5a9c5 --virtualsize 2097152K --name brick_ac730765fa6d170eb90c9795eaf5a9c5
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_ac730765fa6d170eb90c9795eaf5a9c5" created.
>[kubeexec] DEBUG 2018/06/08 08:41:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_d389f0278a774bd7443a09af960961d8/tp_ff2a5406a86cd765269a1b519b76a810 --virtualsize 2097152K --name brick_ff2a5406a86cd765269a1b519b76a810
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_ff2a5406a86cd765269a1b519b76a810" created.
>[kubeexec] DEBUG 2018/06/08 08:41:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_96f1667f2f1ced2c5ef94772922be93b/tp_300c73448ebb5017a216c6661178ca8c --virtualsize 2097152K --name brick_300c73448ebb5017a216c6661178ca8c
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_300c73448ebb5017a216c6661178ca8c" created.
>[kubeexec] DEBUG 2018/06/08 08:41:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_ff2a5406a86cd765269a1b519b76a810
>Result: meta-data=/dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_ff2a5406a86cd765269a1b519b76a810 isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:41:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_ac730765fa6d170eb90c9795eaf5a9c5
>Result: meta-data=/dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_ac730765fa6d170eb90c9795eaf5a9c5 isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:41:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_300c73448ebb5017a216c6661178ca8c
>Result: meta-data=/dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_300c73448ebb5017a216c6661178ca8c isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:41:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: awk "BEGIN {print \"/dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_ff2a5406a86cd765269a1b519b76a810 /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_ff2a5406a86cd765269a1b519b76a810 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:41:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: awk "BEGIN {print \"/dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_ac730765fa6d170eb90c9795eaf5a9c5 /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_ac730765fa6d170eb90c9795eaf5a9c5 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:41:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: awk "BEGIN {print \"/dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_300c73448ebb5017a216c6661178ca8c /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_300c73448ebb5017a216c6661178ca8c xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[negroni] Started GET /queue/e762df464aa012622774563dece1d0c2
>[negroni] Completed 200 OK in 127.075Âµs
>[kubeexec] DEBUG 2018/06/08 08:41:35 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_ff2a5406a86cd765269a1b519b76a810 /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_ff2a5406a86cd765269a1b519b76a810
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:41:35 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_ac730765fa6d170eb90c9795eaf5a9c5 /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_ac730765fa6d170eb90c9795eaf5a9c5
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:41:35 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_300c73448ebb5017a216c6661178ca8c /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_300c73448ebb5017a216c6661178ca8c
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:41:35 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_ff2a5406a86cd765269a1b519b76a810/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:41:35 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_ac730765fa6d170eb90c9795eaf5a9c5/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:41:35 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_300c73448ebb5017a216c6661178ca8c/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:41:35 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: chown :2001 /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_ff2a5406a86cd765269a1b519b76a810/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:41:35 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: chown :2001 /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_ac730765fa6d170eb90c9795eaf5a9c5/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:41:35 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: chown :2001 /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_300c73448ebb5017a216c6661178ca8c/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:41:35 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: chmod 2775 /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_ff2a5406a86cd765269a1b519b76a810/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:41:35 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: chmod 2775 /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_ac730765fa6d170eb90c9795eaf5a9c5/brick
>Result: 
>[negroni] Started GET /queue/e6d8d4b63c9bfa929b21498f22af956e
>[negroni] Completed 200 OK in 103.647Âµs
>[kubeexec] DEBUG 2018/06/08 08:41:35 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: chmod 2775 /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_300c73448ebb5017a216c6661178ca8c/brick
>Result: 
>[cmdexec] INFO 2018/06/08 08:41:35 Creating volume vol_82aebcd425bd89e0d33e6a9bbd1f000d replica 3
>[kubeexec] DEBUG 2018/06/08 08:41:35 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: gluster --mode=script volume create vol_82aebcd425bd89e0d33e6a9bbd1f000d replica 3 10.70.46.187:/var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_ff2a5406a86cd765269a1b519b76a810/brick 10.70.46.122:/var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_ac730765fa6d170eb90c9795eaf5a9c5/brick 10.70.47.76:/var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_300c73448ebb5017a216c6661178ca8c/brick
>Result: volume create: vol_82aebcd425bd89e0d33e6a9bbd1f000d: success: please start the volume to access data
>[kubeexec] DEBUG 2018/06/08 08:41:35 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script volume create vol_b1176cd4e68f9cd8d5ed06cbfd1d12b5 replica 3 10.70.47.76:/var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_4cb2a20a79c4ba5f51a7af73c72792d8/brick 10.70.46.187:/var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_d093b076a37d3e3ba22944bd34ed0f9e/brick 10.70.46.122:/var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_487a6210cd6beafaea7d458f0edc796f/brick
>Result: volume create: vol_b1176cd4e68f9cd8d5ed06cbfd1d12b5: success: please start the volume to access data
>[negroni] Started GET /queue/e762df464aa012622774563dece1d0c2
>[negroni] Completed 200 OK in 122.062Âµs
>[negroni] Started GET /queue/e6d8d4b63c9bfa929b21498f22af956e
>[negroni] Completed 200 OK in 132.67Âµs
>[negroni] Started GET /queue/e762df464aa012622774563dece1d0c2
>[negroni] Completed 200 OK in 104.531Âµs
>[negroni] Started GET /queue/e6d8d4b63c9bfa929b21498f22af956e
>[negroni] Completed 200 OK in 225.152Âµs
>[negroni] Started GET /queue/e762df464aa012622774563dece1d0c2
>[negroni] Completed 200 OK in 114.528Âµs
>[negroni] Started GET /queue/e6d8d4b63c9bfa929b21498f22af956e
>[negroni] Completed 200 OK in 126.61Âµs
>[kubeexec] DEBUG 2018/06/08 08:41:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: gluster --mode=script volume start vol_82aebcd425bd89e0d33e6a9bbd1f000d
>Result: volume start: vol_82aebcd425bd89e0d33e6a9bbd1f000d: success
>[heketi] INFO 2018/06/08 08:41:38 Create Volume succeeded
>[asynchttp] INFO 2018/06/08 08:41:38 asynchttp.go:292: Completed job e6d8d4b63c9bfa929b21498f22af956e in 5.567575808s
>[negroni] Started GET /queue/e762df464aa012622774563dece1d0c2
>[negroni] Completed 200 OK in 124.077Âµs
>[negroni] Started GET /queue/e6d8d4b63c9bfa929b21498f22af956e
>[negroni] Completed 303 See Other in 131.237Âµs
>[negroni] Started GET /volumes/82aebcd425bd89e0d33e6a9bbd1f000d
>[negroni] Completed 200 OK in 3.608472ms
>[negroni] Started GET /clusters/0a73c60efdd4673113b668afea101e6d
>[negroni] Completed 200 OK in 381.366Âµs
>[negroni] Started GET /nodes/278bd6b4e16a8e62ef15aaae22e6abc1
>[negroni] Completed 200 OK in 1.866118ms
>[negroni] Started GET /nodes/70423cc0bcd044fe5ba8bbbd256a3e49
>[negroni] Completed 200 OK in 1.565505ms
>[negroni] Started GET /nodes/d942fe6c0ee5691b7cc263968f97b650
>[negroni] Completed 200 OK in 753.805Âµs
>[negroni] Started GET /queue/e762df464aa012622774563dece1d0c2
>[negroni] Completed 200 OK in 165.641Âµs
>[negroni] Started GET /queue/e762df464aa012622774563dece1d0c2
>[negroni] Completed 200 OK in 134.804Âµs
>[negroni] Started GET /queue/e762df464aa012622774563dece1d0c2
>[negroni] Completed 200 OK in 135.959Âµs
>[kubeexec] DEBUG 2018/06/08 08:41:42 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script volume start vol_b1176cd4e68f9cd8d5ed06cbfd1d12b5
>Result: volume start: vol_b1176cd4e68f9cd8d5ed06cbfd1d12b5: success
>[heketi] INFO 2018/06/08 08:41:42 Create Volume succeeded
>[asynchttp] INFO 2018/06/08 08:41:42 asynchttp.go:292: Completed job e762df464aa012622774563dece1d0c2 in 9.937996305s
>[negroni] Started GET /queue/e762df464aa012622774563dece1d0c2
>[negroni] Completed 303 See Other in 152.851Âµs
>[negroni] Started GET /volumes/b1176cd4e68f9cd8d5ed06cbfd1d12b5
>[negroni] Completed 200 OK in 5.331127ms
>[negroni] Started GET /clusters/0a73c60efdd4673113b668afea101e6d
>[negroni] Completed 200 OK in 218.784Âµs
>[negroni] Started GET /nodes/278bd6b4e16a8e62ef15aaae22e6abc1
>[negroni] Completed 200 OK in 2.644292ms
>[negroni] Started GET /nodes/70423cc0bcd044fe5ba8bbbd256a3e49
>[negroni] Completed 200 OK in 1.247719ms
>[negroni] Started GET /nodes/d942fe6c0ee5691b7cc263968f97b650
>[negroni] Completed 200 OK in 767.683Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 1.065244ms
>[negroni] Started GET /volumes/82aebcd425bd89e0d33e6a9bbd1f000d
>[negroni] Completed 200 OK in 731.218Âµs
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 958.007Âµs
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 1.080655ms
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 816.114Âµs
>[negroni] Started GET /volumes/b1176cd4e68f9cd8d5ed06cbfd1d12b5
>[negroni] Completed 200 OK in 642.058Âµs
>[negroni] Started DELETE /volumes/82aebcd425bd89e0d33e6a9bbd1f000d
>[negroni] Completed 202 Accepted in 13.504844ms
>[asynchttp] INFO 2018/06/08 08:41:45 asynchttp.go:288: Started job 0a4cf62261c075ae3ed7e14a088725a5
>[heketi] INFO 2018/06/08 08:41:45 Started async operation: Delete Volume
>[negroni] Started GET /queue/0a4cf62261c075ae3ed7e14a088725a5
>[negroni] Completed 200 OK in 155.272Âµs
>[negroni] Started DELETE /volumes/b1176cd4e68f9cd8d5ed06cbfd1d12b5
>[negroni] Completed 202 Accepted in 12.556254ms
>[asynchttp] INFO 2018/06/08 08:41:45 asynchttp.go:288: Started job 2ab3a3da6f9986f0807bddf5b0c58529
>[heketi] INFO 2018/06/08 08:41:45 Started async operation: Delete Volume
>[negroni] Started GET /queue/2ab3a3da6f9986f0807bddf5b0c58529
>[negroni] Completed 200 OK in 110.605Âµs
>[kubeexec] DEBUG 2018/06/08 08:41:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script snapshot list vol_82aebcd425bd89e0d33e6a9bbd1f000d --xml
>Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
><cliOutput>
>  <opRet>0</opRet>
>  <opErrno>0</opErrno>
>  <opErrstr/>
>  <snapList>
>    <count>0</count>
>  </snapList>
></cliOutput>
>[kubeexec] DEBUG 2018/06/08 08:41:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script snapshot list vol_b1176cd4e68f9cd8d5ed06cbfd1d12b5 --xml
>Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
><cliOutput>
>  <opRet>0</opRet>
>  <opErrno>0</opErrno>
>  <opErrstr/>
>  <snapList>
>    <count>0</count>
>  </snapList>
></cliOutput>
>[negroni] Started GET /queue/0a4cf62261c075ae3ed7e14a088725a5
>[negroni] Completed 200 OK in 131.927Âµs
>[negroni] Started GET /queue/2ab3a3da6f9986f0807bddf5b0c58529
>[negroni] Completed 200 OK in 168.983Âµs
>[negroni] Started GET /queue/0a4cf62261c075ae3ed7e14a088725a5
>[negroni] Completed 200 OK in 190.251Âµs
>[negroni] Started GET /queue/2ab3a3da6f9986f0807bddf5b0c58529
>[negroni] Completed 200 OK in 137.632Âµs
>[kubeexec] DEBUG 2018/06/08 08:41:48 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script volume stop vol_82aebcd425bd89e0d33e6a9bbd1f000d force
>Result: volume stop: vol_82aebcd425bd89e0d33e6a9bbd1f000d: success
>[negroni] Started GET /queue/0a4cf62261c075ae3ed7e14a088725a5
>[negroni] Completed 200 OK in 120.432Âµs
>[negroni] Started GET /queue/2ab3a3da6f9986f0807bddf5b0c58529
>[negroni] Completed 200 OK in 119.583Âµs
>[negroni] Started GET /queue/0a4cf62261c075ae3ed7e14a088725a5
>[negroni] Completed 200 OK in 172.83Âµs
>[negroni] Started GET /queue/2ab3a3da6f9986f0807bddf5b0c58529
>[negroni] Completed 200 OK in 198.654Âµs
>[negroni] Started GET /queue/0a4cf62261c075ae3ed7e14a088725a5
>[negroni] Completed 200 OK in 199.305Âµs
>[negroni] Started GET /queue/2ab3a3da6f9986f0807bddf5b0c58529
>[negroni] Completed 200 OK in 192.032Âµs
>[negroni] Started GET /queue/0a4cf62261c075ae3ed7e14a088725a5
>[negroni] Completed 200 OK in 251.152Âµs
>[negroni] Started GET /queue/2ab3a3da6f9986f0807bddf5b0c58529
>[negroni] Completed 200 OK in 224.679Âµs
>[kubeexec] DEBUG 2018/06/08 08:41:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script volume stop vol_b1176cd4e68f9cd8d5ed06cbfd1d12b5 force
>Result: volume stop: vol_b1176cd4e68f9cd8d5ed06cbfd1d12b5: success
>[kubeexec] DEBUG 2018/06/08 08:41:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script volume delete vol_82aebcd425bd89e0d33e6a9bbd1f000d
>Result: volume delete: vol_82aebcd425bd89e0d33e6a9bbd1f000d: success
>[heketi] INFO 2018/06/08 08:41:52 Deleting brick ff2a5406a86cd765269a1b519b76a810
>[heketi] INFO 2018/06/08 08:41:52 Deleting brick 300c73448ebb5017a216c6661178ca8c
>[heketi] INFO 2018/06/08 08:41:52 Deleting brick ac730765fa6d170eb90c9795eaf5a9c5
>[kubeexec] DEBUG 2018/06/08 08:41:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mount | grep -w /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_ff2a5406a86cd765269a1b519b76a810 | cut -d" " -f1
>Result: /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_ff2a5406a86cd765269a1b519b76a810
>[kubeexec] DEBUG 2018/06/08 08:41:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mount | grep -w /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_300c73448ebb5017a216c6661178ca8c | cut -d" " -f1
>Result: /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_300c73448ebb5017a216c6661178ca8c
>[kubeexec] DEBUG 2018/06/08 08:41:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_ff2a5406a86cd765269a1b519b76a810
>
>Result:   vg_d389f0278a774bd7443a09af960961d8/tp_ff2a5406a86cd765269a1b519b76a810
>[kubeexec] DEBUG 2018/06/08 08:41:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script volume delete vol_b1176cd4e68f9cd8d5ed06cbfd1d12b5
>Result: volume delete: vol_b1176cd4e68f9cd8d5ed06cbfd1d12b5: success
>[heketi] INFO 2018/06/08 08:41:52 Deleting brick 4cb2a20a79c4ba5f51a7af73c72792d8
>[heketi] INFO 2018/06/08 08:41:52 Deleting brick d093b076a37d3e3ba22944bd34ed0f9e
>[heketi] INFO 2018/06/08 08:41:52 Deleting brick 487a6210cd6beafaea7d458f0edc796f
>[kubeexec] DEBUG 2018/06/08 08:41:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_300c73448ebb5017a216c6661178ca8c
>
>Result:   vg_96f1667f2f1ced2c5ef94772922be93b/tp_300c73448ebb5017a216c6661178ca8c
>[negroni] Started GET /queue/0a4cf62261c075ae3ed7e14a088725a5
>[negroni] Completed 200 OK in 107.634Âµs
>[kubeexec] DEBUG 2018/06/08 08:41:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mount | grep -w /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_ac730765fa6d170eb90c9795eaf5a9c5 | cut -d" " -f1
>Result: /dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_ac730765fa6d170eb90c9795eaf5a9c5
>[negroni] Started GET /queue/2ab3a3da6f9986f0807bddf5b0c58529
>[negroni] Completed 200 OK in 140.105Âµs
>[kubeexec] DEBUG 2018/06/08 08:41:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: umount /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_ff2a5406a86cd765269a1b519b76a810
>Result: 
>[negroni] Started GET /queue/0a4cf62261c075ae3ed7e14a088725a5
>[negroni] Completed 200 OK in 277.508Âµs
>[kubeexec] DEBUG 2018/06/08 08:41:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mount | grep -w /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_4cb2a20a79c4ba5f51a7af73c72792d8 | cut -d" " -f1
>Result: /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_4cb2a20a79c4ba5f51a7af73c72792d8
>[negroni] Started GET /queue/2ab3a3da6f9986f0807bddf5b0c58529
>[negroni] Completed 200 OK in 138.757Âµs
>[kubeexec] DEBUG 2018/06/08 08:41:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mount | grep -w /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_487a6210cd6beafaea7d458f0edc796f | cut -d" " -f1
>Result: /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_487a6210cd6beafaea7d458f0edc796f
>[kubeexec] DEBUG 2018/06/08 08:41:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mount | grep -w /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_d093b076a37d3e3ba22944bd34ed0f9e | cut -d" " -f1
>Result: /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_d093b076a37d3e3ba22944bd34ed0f9e
>[negroni] Started GET /queue/0a4cf62261c075ae3ed7e14a088725a5
>[negroni] Completed 200 OK in 171.691Âµs
>[negroni] Started GET /queue/2ab3a3da6f9986f0807bddf5b0c58529
>[negroni] Completed 200 OK in 131.176Âµs
>[kubeexec] DEBUG 2018/06/08 08:41:55 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: umount /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_300c73448ebb5017a216c6661178ca8c
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:41:55 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_ac730765fa6d170eb90c9795eaf5a9c5
>
>Result:   vg_66a9af9f6bac95f9d8d556a2f14c29d3/tp_ac730765fa6d170eb90c9795eaf5a9c5
>[kubeexec] DEBUG 2018/06/08 08:41:55 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: sed -i.save "/brick_ff2a5406a86cd765269a1b519b76a810/d" /var/lib/heketi/fstab
>Result: 
>[negroni] Started GET /queue/0a4cf62261c075ae3ed7e14a088725a5
>[negroni] Completed 200 OK in 217.844Âµs
>[negroni] Started GET /queue/2ab3a3da6f9986f0807bddf5b0c58529
>[negroni] Completed 200 OK in 194.602Âµs
>[kubeexec] DEBUG 2018/06/08 08:41:56 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_4cb2a20a79c4ba5f51a7af73c72792d8
>
>Result:   vg_96f1667f2f1ced2c5ef94772922be93b/tp_4cb2a20a79c4ba5f51a7af73c72792d8
>[kubeexec] DEBUG 2018/06/08 08:41:56 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_487a6210cd6beafaea7d458f0edc796f
>
>Result:   vg_3a4297677881963e3f80124971d50eea/tp_487a6210cd6beafaea7d458f0edc796f
>[kubeexec] DEBUG 2018/06/08 08:41:56 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_d093b076a37d3e3ba22944bd34ed0f9e
>
>Result:   vg_b861af9cc67f5bf8d5a8f9e38f56b14e/tp_d093b076a37d3e3ba22944bd34ed0f9e
>[negroni] Started GET /queue/0a4cf62261c075ae3ed7e14a088725a5
>[negroni] Completed 200 OK in 156.197Âµs
>[negroni] Started GET /queue/2ab3a3da6f9986f0807bddf5b0c58529
>[negroni] Completed 200 OK in 170.044Âµs
>[kubeexec] DEBUG 2018/06/08 08:41:57 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: sed -i.save "/brick_300c73448ebb5017a216c6661178ca8c/d" /var/lib/heketi/fstab
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:41:57 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: umount /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_ac730765fa6d170eb90c9795eaf5a9c5
>Result: 
>[negroni] Started GET /queue/0a4cf62261c075ae3ed7e14a088725a5
>[negroni] Completed 200 OK in 222.322Âµs
>[kubeexec] DEBUG 2018/06/08 08:41:57 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvremove --autobackup=n -f /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_ff2a5406a86cd765269a1b519b76a810
>
>Result:   Logical volume "brick_ff2a5406a86cd765269a1b519b76a810" successfully removed
>[negroni] Started GET /queue/2ab3a3da6f9986f0807bddf5b0c58529
>[negroni] Completed 200 OK in 125.286Âµs
>[kubeexec] DEBUG 2018/06/08 08:41:58 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: umount /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_4cb2a20a79c4ba5f51a7af73c72792d8
>Result: 
>[negroni] Started GET /queue/0a4cf62261c075ae3ed7e14a088725a5
>[negroni] Completed 200 OK in 186.37Âµs
>[kubeexec] DEBUG 2018/06/08 08:41:58 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: umount /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_487a6210cd6beafaea7d458f0edc796f
>Result: 
>[negroni] Started GET /queue/2ab3a3da6f9986f0807bddf5b0c58529
>[negroni] Completed 200 OK in 145.366Âµs
>[kubeexec] DEBUG 2018/06/08 08:41:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: umount /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_d093b076a37d3e3ba22944bd34ed0f9e
>Result: 
>[negroni] Started GET /queue/0a4cf62261c075ae3ed7e14a088725a5
>[negroni] Completed 200 OK in 170.982Âµs
>[negroni] Started GET /queue/2ab3a3da6f9986f0807bddf5b0c58529
>[negroni] Completed 200 OK in 178.699Âµs
>[kubeexec] DEBUG 2018/06/08 08:41:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvremove --autobackup=n -f /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_300c73448ebb5017a216c6661178ca8c
>
>Result:   Logical volume "brick_300c73448ebb5017a216c6661178ca8c" successfully removed
>[kubeexec] DEBUG 2018/06/08 08:42:00 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: sed -i.save "/brick_ac730765fa6d170eb90c9795eaf5a9c5/d" /var/lib/heketi/fstab
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:42:00 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvs --noheadings --options=thin_count   vg_d389f0278a774bd7443a09af960961d8/tp_ff2a5406a86cd765269a1b519b76a810
>
>Result:        0
>[negroni] Started GET /queue/0a4cf62261c075ae3ed7e14a088725a5
>[negroni] Completed 200 OK in 144.645Âµs
>[negroni] Started GET /queue/2ab3a3da6f9986f0807bddf5b0c58529
>[negroni] Completed 200 OK in 180.43Âµs
>[kubeexec] DEBUG 2018/06/08 08:42:01 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: sed -i.save "/brick_4cb2a20a79c4ba5f51a7af73c72792d8/d" /var/lib/heketi/fstab
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:42:01 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: sed -i.save "/brick_487a6210cd6beafaea7d458f0edc796f/d" /var/lib/heketi/fstab
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:42:01 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: sed -i.save "/brick_d093b076a37d3e3ba22944bd34ed0f9e/d" /var/lib/heketi/fstab
>Result: 
>[negroni] Started GET /queue/0a4cf62261c075ae3ed7e14a088725a5
>[negroni] Completed 200 OK in 142.815Âµs
>[negroni] Started GET /queue/2ab3a3da6f9986f0807bddf5b0c58529
>[negroni] Completed 200 OK in 268.898Âµs
>[kubeexec] DEBUG 2018/06/08 08:42:02 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvs --noheadings --options=thin_count   vg_96f1667f2f1ced2c5ef94772922be93b/tp_300c73448ebb5017a216c6661178ca8c
>
>Result:        0
>[kubeexec] DEBUG 2018/06/08 08:42:02 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvremove --autobackup=n -f /dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_ac730765fa6d170eb90c9795eaf5a9c5
>
>Result:   Logical volume "brick_ac730765fa6d170eb90c9795eaf5a9c5" successfully removed
>[negroni] Started GET /queue/0a4cf62261c075ae3ed7e14a088725a5
>[negroni] Completed 200 OK in 132.044Âµs
>[kubeexec] DEBUG 2018/06/08 08:42:02 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvremove --autobackup=n -f   vg_d389f0278a774bd7443a09af960961d8/tp_ff2a5406a86cd765269a1b519b76a810
>
>Result:   Logical volume "tp_ff2a5406a86cd765269a1b519b76a810" successfully removed
>[negroni] Started GET /queue/2ab3a3da6f9986f0807bddf5b0c58529
>[negroni] Completed 200 OK in 184.758Âµs
>[kubeexec] DEBUG 2018/06/08 08:42:03 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvremove --autobackup=n -f /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_4cb2a20a79c4ba5f51a7af73c72792d8
>
>Result:   Logical volume "brick_4cb2a20a79c4ba5f51a7af73c72792d8" successfully removed
>[kubeexec] DEBUG 2018/06/08 08:42:03 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvremove --autobackup=n -f /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_487a6210cd6beafaea7d458f0edc796f
>
>Result:   Logical volume "brick_487a6210cd6beafaea7d458f0edc796f" successfully removed
>[negroni] Started GET /queue/0a4cf62261c075ae3ed7e14a088725a5
>[negroni] Completed 200 OK in 137.745Âµs
>[kubeexec] DEBUG 2018/06/08 08:42:03 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvremove --autobackup=n -f /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_d093b076a37d3e3ba22944bd34ed0f9e
>
>Result:   Logical volume "brick_d093b076a37d3e3ba22944bd34ed0f9e" successfully removed
>[negroni] Started GET /queue/2ab3a3da6f9986f0807bddf5b0c58529
>[negroni] Completed 200 OK in 186.925Âµs
>[negroni] Started GET /queue/0a4cf62261c075ae3ed7e14a088725a5
>[negroni] Completed 200 OK in 139.991Âµs
>[kubeexec] DEBUG 2018/06/08 08:42:04 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvremove --autobackup=n -f   vg_96f1667f2f1ced2c5ef94772922be93b/tp_300c73448ebb5017a216c6661178ca8c
>
>Result:   Logical volume "tp_300c73448ebb5017a216c6661178ca8c" successfully removed
>[kubeexec] DEBUG 2018/06/08 08:42:04 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvs --noheadings --options=thin_count   vg_66a9af9f6bac95f9d8d556a2f14c29d3/tp_ac730765fa6d170eb90c9795eaf5a9c5
>
>Result:        0
>[negroni] Started GET /queue/2ab3a3da6f9986f0807bddf5b0c58529
>[negroni] Completed 200 OK in 125.317Âµs
>[kubeexec] DEBUG 2018/06/08 08:42:05 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: rmdir /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_ff2a5406a86cd765269a1b519b76a810
>Result: 
>[negroni] Started GET /queue/0a4cf62261c075ae3ed7e14a088725a5
>[negroni] Completed 200 OK in 169.632Âµs
>[kubeexec] DEBUG 2018/06/08 08:42:05 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvs --noheadings --options=thin_count   vg_96f1667f2f1ced2c5ef94772922be93b/tp_4cb2a20a79c4ba5f51a7af73c72792d8
>
>Result:        0
>[negroni] Started GET /queue/2ab3a3da6f9986f0807bddf5b0c58529
>[negroni] Completed 200 OK in 133.509Âµs
>[kubeexec] DEBUG 2018/06/08 08:42:06 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvs --noheadings --options=thin_count   vg_3a4297677881963e3f80124971d50eea/tp_487a6210cd6beafaea7d458f0edc796f
>
>Result:        0
>[kubeexec] DEBUG 2018/06/08 08:42:06 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvs --noheadings --options=thin_count   vg_b861af9cc67f5bf8d5a8f9e38f56b14e/tp_d093b076a37d3e3ba22944bd34ed0f9e
>
>Result:        0
>[negroni] Started GET /queue/0a4cf62261c075ae3ed7e14a088725a5
>[negroni] Completed 200 OK in 178.672Âµs
>[negroni] Started GET /queue/2ab3a3da6f9986f0807bddf5b0c58529
>[negroni] Completed 200 OK in 207.046Âµs
>[kubeexec] DEBUG 2018/06/08 08:42:07 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: rmdir /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_300c73448ebb5017a216c6661178ca8c
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:42:07 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvremove --autobackup=n -f   vg_66a9af9f6bac95f9d8d556a2f14c29d3/tp_ac730765fa6d170eb90c9795eaf5a9c5
>
>Result:   Logical volume "tp_ac730765fa6d170eb90c9795eaf5a9c5" successfully removed
>[kubeexec] DEBUG 2018/06/08 08:42:07 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvremove --autobackup=n -f   vg_b861af9cc67f5bf8d5a8f9e38f56b14e/tp_d093b076a37d3e3ba22944bd34ed0f9e
>
>Result:   Logical volume "tp_d093b076a37d3e3ba22944bd34ed0f9e" successfully removed
>[negroni] Started GET /queue/0a4cf62261c075ae3ed7e14a088725a5
>[negroni] Completed 200 OK in 157.232Âµs
>[negroni] Started GET /queue/2ab3a3da6f9986f0807bddf5b0c58529
>[negroni] Completed 200 OK in 127.264Âµs
>[kubeexec] DEBUG 2018/06/08 08:42:08 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvremove --autobackup=n -f   vg_96f1667f2f1ced2c5ef94772922be93b/tp_4cb2a20a79c4ba5f51a7af73c72792d8
>
>Result:   Logical volume "tp_4cb2a20a79c4ba5f51a7af73c72792d8" successfully removed
>[kubeexec] DEBUG 2018/06/08 08:42:08 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvremove --autobackup=n -f   vg_3a4297677881963e3f80124971d50eea/tp_487a6210cd6beafaea7d458f0edc796f
>
>Result:   Logical volume "tp_487a6210cd6beafaea7d458f0edc796f" successfully removed
>[kubeexec] DEBUG 2018/06/08 08:42:08 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: rmdir /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_d093b076a37d3e3ba22944bd34ed0f9e
>Result: 
>[negroni] Started GET /queue/0a4cf62261c075ae3ed7e14a088725a5
>[negroni] Completed 200 OK in 194.842Âµs
>[negroni] Started GET /queue/2ab3a3da6f9986f0807bddf5b0c58529
>[negroni] Completed 200 OK in 183.655Âµs
>[kubeexec] DEBUG 2018/06/08 08:42:09 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: rmdir /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_4cb2a20a79c4ba5f51a7af73c72792d8
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:42:09 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: rmdir /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_ac730765fa6d170eb90c9795eaf5a9c5
>Result: 
>[heketi] INFO 2018/06/08 08:42:09 Delete Volume succeeded
>[asynchttp] INFO 2018/06/08 08:42:09 asynchttp.go:292: Completed job 0a4cf62261c075ae3ed7e14a088725a5 in 23.781212579s
>[negroni] Started GET /queue/0a4cf62261c075ae3ed7e14a088725a5
>[negroni] Completed 204 No Content in 184.386Âµs
>[negroni] Started DELETE /volumes/82aebcd425bd89e0d33e6a9bbd1f000d
>[negroni] Completed 404 Not Found in 2.48331ms
>[kubeexec] DEBUG 2018/06/08 08:42:09 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: rmdir /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_487a6210cd6beafaea7d458f0edc796f
>Result: 
>[heketi] INFO 2018/06/08 08:42:09 Delete Volume succeeded
>[asynchttp] INFO 2018/06/08 08:42:09 asynchttp.go:292: Completed job 2ab3a3da6f9986f0807bddf5b0c58529 in 23.980774213s
>[negroni] Started GET /queue/2ab3a3da6f9986f0807bddf5b0c58529
>[negroni] Completed 204 No Content in 191.255Âµs
>[negroni] Started DELETE /volumes/b1176cd4e68f9cd8d5ed06cbfd1d12b5
>[negroni] Completed 404 Not Found in 2.078887ms
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 199.768Âµs
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 2.758047ms
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 604.138Âµs
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 1.066534ms
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 236.192Âµs
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 679.256Âµs
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 595.592Âµs
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 513.18Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 181.753Âµs
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 634.022Âµs
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 744.733Âµs
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 625.994Âµs
>[negroni] Started POST /volumes
>[heketi] INFO 2018/06/08 08:42:10 Allocating brick set #0
>[negroni] Completed 202 Accepted in 13.369515ms
>[asynchttp] INFO 2018/06/08 08:42:11 asynchttp.go:288: Started job 59515cc633de3ff6e62095c3ef71ec7b
>[heketi] INFO 2018/06/08 08:42:11 Started async operation: Create Volume
>[negroni] Started GET /queue/59515cc633de3ff6e62095c3ef71ec7b
>[negroni] Completed 200 OK in 105.948Âµs
>[heketi] INFO 2018/06/08 08:42:11 Creating brick d7770ed15229b256bf58c081ad57ea73
>[heketi] INFO 2018/06/08 08:42:11 Creating brick e5ae8fc12a741a851b2aade547fe3615
>[heketi] INFO 2018/06/08 08:42:11 Creating brick bfb72d4e8e105df407a890930c475175
>[kubeexec] DEBUG 2018/06/08 08:42:11 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir -p /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_e5ae8fc12a741a851b2aade547fe3615
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:42:11 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir -p /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_d7770ed15229b256bf58c081ad57ea73
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:42:11 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir -p /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_bfb72d4e8e105df407a890930c475175
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:42:11 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_66a9af9f6bac95f9d8d556a2f14c29d3/tp_bfb72d4e8e105df407a890930c475175 --virtualsize 2097152K --name brick_bfb72d4e8e105df407a890930c475175
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_bfb72d4e8e105df407a890930c475175" created.
>[kubeexec] DEBUG 2018/06/08 08:42:11 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_96f1667f2f1ced2c5ef94772922be93b/tp_e5ae8fc12a741a851b2aade547fe3615 --virtualsize 2097152K --name brick_e5ae8fc12a741a851b2aade547fe3615
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_e5ae8fc12a741a851b2aade547fe3615" created.
>[kubeexec] DEBUG 2018/06/08 08:42:11 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_d389f0278a774bd7443a09af960961d8/tp_d7770ed15229b256bf58c081ad57ea73 --virtualsize 2097152K --name brick_d7770ed15229b256bf58c081ad57ea73
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_d7770ed15229b256bf58c081ad57ea73" created.
>[kubeexec] DEBUG 2018/06/08 08:42:11 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_bfb72d4e8e105df407a890930c475175
>Result: meta-data=/dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_bfb72d4e8e105df407a890930c475175 isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:42:11 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_e5ae8fc12a741a851b2aade547fe3615
>Result: meta-data=/dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_e5ae8fc12a741a851b2aade547fe3615 isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:42:11 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_d7770ed15229b256bf58c081ad57ea73
>Result: meta-data=/dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_d7770ed15229b256bf58c081ad57ea73 isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:42:11 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: awk "BEGIN {print \"/dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_bfb72d4e8e105df407a890930c475175 /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_bfb72d4e8e105df407a890930c475175 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:42:11 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: awk "BEGIN {print \"/dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_d7770ed15229b256bf58c081ad57ea73 /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_d7770ed15229b256bf58c081ad57ea73 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:42:11 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: awk "BEGIN {print \"/dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_e5ae8fc12a741a851b2aade547fe3615 /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_e5ae8fc12a741a851b2aade547fe3615 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:42:11 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_bfb72d4e8e105df407a890930c475175 /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_bfb72d4e8e105df407a890930c475175
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:42:11 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_d7770ed15229b256bf58c081ad57ea73 /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_d7770ed15229b256bf58c081ad57ea73
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:42:11 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_e5ae8fc12a741a851b2aade547fe3615 /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_e5ae8fc12a741a851b2aade547fe3615
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:42:11 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_bfb72d4e8e105df407a890930c475175/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:42:11 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_e5ae8fc12a741a851b2aade547fe3615/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:42:11 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_d7770ed15229b256bf58c081ad57ea73/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:42:11 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: chown :2000 /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_bfb72d4e8e105df407a890930c475175/brick
>Result: 
>[negroni] Started GET /queue/59515cc633de3ff6e62095c3ef71ec7b
>[negroni] Completed 200 OK in 103.52Âµs
>[kubeexec] DEBUG 2018/06/08 08:42:12 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: chown :2000 /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_d7770ed15229b256bf58c081ad57ea73/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:42:12 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: chown :2000 /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_e5ae8fc12a741a851b2aade547fe3615/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:42:12 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: chmod 2775 /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_bfb72d4e8e105df407a890930c475175/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:42:12 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: chmod 2775 /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_d7770ed15229b256bf58c081ad57ea73/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:42:12 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: chmod 2775 /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_e5ae8fc12a741a851b2aade547fe3615/brick
>Result: 
>[cmdexec] INFO 2018/06/08 08:42:12 Creating volume vol_7851878e4341c93e517bff5cad8f7b49 replica 3
>[kubeexec] DEBUG 2018/06/08 08:42:12 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script volume create vol_7851878e4341c93e517bff5cad8f7b49 replica 3 10.70.46.122:/var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_bfb72d4e8e105df407a890930c475175/brick 10.70.47.76:/var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_e5ae8fc12a741a851b2aade547fe3615/brick 10.70.46.187:/var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_d7770ed15229b256bf58c081ad57ea73/brick
>Result: volume create: vol_7851878e4341c93e517bff5cad8f7b49: success: please start the volume to access data
>[negroni] Started GET /queue/59515cc633de3ff6e62095c3ef71ec7b
>[negroni] Completed 200 OK in 114.191Âµs
>[negroni] Started GET /queue/59515cc633de3ff6e62095c3ef71ec7b
>[negroni] Completed 200 OK in 121.978Âµs
>[negroni] Started GET /queue/59515cc633de3ff6e62095c3ef71ec7b
>[negroni] Completed 200 OK in 118.228Âµs
>[negroni] Started GET /queue/59515cc633de3ff6e62095c3ef71ec7b
>[negroni] Completed 200 OK in 132.456Âµs
>[kubeexec] DEBUG 2018/06/08 08:42:16 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script volume start vol_7851878e4341c93e517bff5cad8f7b49
>Result: volume start: vol_7851878e4341c93e517bff5cad8f7b49: success
>[heketi] INFO 2018/06/08 08:42:16 Create Volume succeeded
>[asynchttp] INFO 2018/06/08 08:42:16 asynchttp.go:292: Completed job 59515cc633de3ff6e62095c3ef71ec7b in 5.62399169s
>[negroni] Started GET /queue/59515cc633de3ff6e62095c3ef71ec7b
>[negroni] Completed 303 See Other in 139.381Âµs
>[negroni] Started GET /volumes/7851878e4341c93e517bff5cad8f7b49
>[negroni] Completed 200 OK in 3.177048ms
>[negroni] Started GET /clusters/0a73c60efdd4673113b668afea101e6d
>[negroni] Completed 200 OK in 181.198Âµs
>[negroni] Started GET /nodes/278bd6b4e16a8e62ef15aaae22e6abc1
>[negroni] Completed 200 OK in 2.306593ms
>[negroni] Started GET /nodes/70423cc0bcd044fe5ba8bbbd256a3e49
>[negroni] Completed 200 OK in 1.050894ms
>[negroni] Started GET /nodes/d942fe6c0ee5691b7cc263968f97b650
>[negroni] Completed 200 OK in 965.488Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 1.655388ms
>[negroni] Started GET /volumes/7851878e4341c93e517bff5cad8f7b49
>[negroni] Completed 200 OK in 1.184758ms
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 579.775Âµs
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 556.564Âµs
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 563.353Âµs
>[negroni] Started POST /volumes
>[heketi] INFO 2018/06/08 08:42:18 Allocating brick set #0
>[negroni] Completed 202 Accepted in 12.427785ms
>[asynchttp] INFO 2018/06/08 08:42:18 asynchttp.go:288: Started job 45e38b9bc2afc427093a8aff2f3898c1
>[heketi] INFO 2018/06/08 08:42:18 Started async operation: Create Volume
>[negroni] Started GET /queue/45e38b9bc2afc427093a8aff2f3898c1
>[negroni] Completed 200 OK in 98.937Âµs
>[heketi] INFO 2018/06/08 08:42:18 Creating brick dc9c085778d516278c0cadca4db4be15
>[heketi] INFO 2018/06/08 08:42:18 Creating brick 5b2e140dcb360bbba091b9c24633aec7
>[heketi] INFO 2018/06/08 08:42:18 Creating brick 4c77a98309b18630c0618c89c97ba0e0
>[kubeexec] DEBUG 2018/06/08 08:42:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir -p /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_4c77a98309b18630c0618c89c97ba0e0
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:42:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir -p /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_dc9c085778d516278c0cadca4db4be15
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:42:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir -p /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_5b2e140dcb360bbba091b9c24633aec7
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:42:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_3a4297677881963e3f80124971d50eea/tp_5b2e140dcb360bbba091b9c24633aec7 --virtualsize 2097152K --name brick_5b2e140dcb360bbba091b9c24633aec7
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_5b2e140dcb360bbba091b9c24633aec7" created.
>[kubeexec] DEBUG 2018/06/08 08:42:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_9394bc70699b006c5460c9f654cf345f/tp_4c77a98309b18630c0618c89c97ba0e0 --virtualsize 2097152K --name brick_4c77a98309b18630c0618c89c97ba0e0
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_4c77a98309b18630c0618c89c97ba0e0" created.
>[kubeexec] DEBUG 2018/06/08 08:42:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_d389f0278a774bd7443a09af960961d8/tp_dc9c085778d516278c0cadca4db4be15 --virtualsize 2097152K --name brick_dc9c085778d516278c0cadca4db4be15
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_dc9c085778d516278c0cadca4db4be15" created.
>[kubeexec] DEBUG 2018/06/08 08:42:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_5b2e140dcb360bbba091b9c24633aec7
>Result: meta-data=/dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_5b2e140dcb360bbba091b9c24633aec7 isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:42:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_dc9c085778d516278c0cadca4db4be15
>Result: meta-data=/dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_dc9c085778d516278c0cadca4db4be15 isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:42:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: awk "BEGIN {print \"/dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_5b2e140dcb360bbba091b9c24633aec7 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_5b2e140dcb360bbba091b9c24633aec7 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:42:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: awk "BEGIN {print \"/dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_dc9c085778d516278c0cadca4db4be15 /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_dc9c085778d516278c0cadca4db4be15 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:42:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_4c77a98309b18630c0618c89c97ba0e0
>Result: meta-data=/dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_4c77a98309b18630c0618c89c97ba0e0 isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:42:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: awk "BEGIN {print \"/dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_4c77a98309b18630c0618c89c97ba0e0 /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_4c77a98309b18630c0618c89c97ba0e0 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:42:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_dc9c085778d516278c0cadca4db4be15 /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_dc9c085778d516278c0cadca4db4be15
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:42:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_5b2e140dcb360bbba091b9c24633aec7 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_5b2e140dcb360bbba091b9c24633aec7
>Result: 
>[negroni] Started GET /queue/45e38b9bc2afc427093a8aff2f3898c1
>[negroni] Completed 200 OK in 178.119Âµs
>[kubeexec] DEBUG 2018/06/08 08:42:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_5b2e140dcb360bbba091b9c24633aec7/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:42:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_dc9c085778d516278c0cadca4db4be15/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:42:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_4c77a98309b18630c0618c89c97ba0e0 /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_4c77a98309b18630c0618c89c97ba0e0
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:42:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: chown :2001 /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_dc9c085778d516278c0cadca4db4be15/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:42:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: chown :2001 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_5b2e140dcb360bbba091b9c24633aec7/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:42:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_4c77a98309b18630c0618c89c97ba0e0/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:42:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: chmod 2775 /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_dc9c085778d516278c0cadca4db4be15/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:42:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: chmod 2775 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_5b2e140dcb360bbba091b9c24633aec7/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:42:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: chown :2001 /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_4c77a98309b18630c0618c89c97ba0e0/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:42:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: chmod 2775 /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_4c77a98309b18630c0618c89c97ba0e0/brick
>Result: 
>[cmdexec] INFO 2018/06/08 08:42:19 Creating volume vol_48f9db6f6531b638a436926286f6eefb replica 3
>[kubeexec] DEBUG 2018/06/08 08:42:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script volume create vol_48f9db6f6531b638a436926286f6eefb replica 3 10.70.46.122:/var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_5b2e140dcb360bbba091b9c24633aec7/brick 10.70.47.76:/var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_4c77a98309b18630c0618c89c97ba0e0/brick 10.70.46.187:/var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_dc9c085778d516278c0cadca4db4be15/brick
>Result: volume create: vol_48f9db6f6531b638a436926286f6eefb: success: please start the volume to access data
>[negroni] Started GET /queue/45e38b9bc2afc427093a8aff2f3898c1
>[negroni] Completed 200 OK in 166.343Âµs
>[negroni] Started GET /queue/45e38b9bc2afc427093a8aff2f3898c1
>[negroni] Completed 200 OK in 144.698Âµs
>[negroni] Started GET /queue/45e38b9bc2afc427093a8aff2f3898c1
>[negroni] Completed 200 OK in 140.676Âµs
>[kubeexec] DEBUG 2018/06/08 08:42:23 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script volume start vol_48f9db6f6531b638a436926286f6eefb
>Result: volume start: vol_48f9db6f6531b638a436926286f6eefb: success
>[heketi] INFO 2018/06/08 08:42:23 Create Volume succeeded
>[asynchttp] INFO 2018/06/08 08:42:23 asynchttp.go:292: Completed job 45e38b9bc2afc427093a8aff2f3898c1 in 4.808623056s
>[negroni] Started GET /queue/45e38b9bc2afc427093a8aff2f3898c1
>[negroni] Completed 303 See Other in 257.988Âµs
>[negroni] Started GET /volumes/48f9db6f6531b638a436926286f6eefb
>[negroni] Completed 200 OK in 5.975968ms
>[negroni] Started GET /clusters/0a73c60efdd4673113b668afea101e6d
>[negroni] Completed 200 OK in 398.081Âµs
>[negroni] Started GET /nodes/278bd6b4e16a8e62ef15aaae22e6abc1
>[negroni] Completed 200 OK in 4.60131ms
>[negroni] Started GET /nodes/70423cc0bcd044fe5ba8bbbd256a3e49
>[negroni] Completed 200 OK in 2.047219ms
>[negroni] Started GET /nodes/d942fe6c0ee5691b7cc263968f97b650
>[negroni] Completed 200 OK in 1.35698ms
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 848.237Âµs
>[negroni] Started GET /volumes/48f9db6f6531b638a436926286f6eefb
>[negroni] Completed 200 OK in 610.21Âµs
>[negroni] Started GET /volumes/7851878e4341c93e517bff5cad8f7b49
>[negroni] Completed 200 OK in 544.35Âµs
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 620.488Âµs
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 834.869Âµs
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 1.791059ms
>[negroni] Started DELETE /volumes/48f9db6f6531b638a436926286f6eefb
>[negroni] Completed 202 Accepted in 8.269575ms
>[asynchttp] INFO 2018/06/08 08:42:25 asynchttp.go:288: Started job 84472ae559c45c9b84ab85d4a5852f91
>[heketi] INFO 2018/06/08 08:42:25 Started async operation: Delete Volume
>[negroni] Started GET /queue/84472ae559c45c9b84ab85d4a5852f91
>[negroni] Completed 200 OK in 135.666Âµs
>[kubeexec] DEBUG 2018/06/08 08:42:25 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script snapshot list vol_48f9db6f6531b638a436926286f6eefb --xml
>Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
><cliOutput>
>  <opRet>0</opRet>
>  <opErrno>0</opErrno>
>  <opErrstr/>
>  <snapList>
>    <count>0</count>
>  </snapList>
></cliOutput>
>[negroni] Started DELETE /volumes/7851878e4341c93e517bff5cad8f7b49
>[negroni] Completed 202 Accepted in 11.408925ms
>[asynchttp] INFO 2018/06/08 08:42:25 asynchttp.go:288: Started job f1e044bedf7f37fb57361f115a62841b
>[heketi] INFO 2018/06/08 08:42:25 Started async operation: Delete Volume
>[negroni] Started GET /queue/f1e044bedf7f37fb57361f115a62841b
>[negroni] Completed 200 OK in 94.73Âµs
>[kubeexec] DEBUG 2018/06/08 08:42:26 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script snapshot list vol_7851878e4341c93e517bff5cad8f7b49 --xml
>Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
><cliOutput>
>  <opRet>0</opRet>
>  <opErrno>0</opErrno>
>  <opErrstr/>
>  <snapList>
>    <count>0</count>
>  </snapList>
></cliOutput>
>[negroni] Started GET /queue/84472ae559c45c9b84ab85d4a5852f91
>[negroni] Completed 200 OK in 167.676Âµs
>[negroni] Started GET /queue/f1e044bedf7f37fb57361f115a62841b
>[negroni] Completed 200 OK in 144.229Âµs
>[negroni] Started GET /queue/84472ae559c45c9b84ab85d4a5852f91
>[negroni] Completed 200 OK in 146.918Âµs
>[negroni] Started GET /queue/f1e044bedf7f37fb57361f115a62841b
>[negroni] Completed 200 OK in 116.417Âµs
>[kubeexec] DEBUG 2018/06/08 08:42:28 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script volume stop vol_48f9db6f6531b638a436926286f6eefb force
>Result: volume stop: vol_48f9db6f6531b638a436926286f6eefb: success
>[negroni] Started GET /queue/84472ae559c45c9b84ab85d4a5852f91
>[negroni] Completed 200 OK in 116.47Âµs
>[negroni] Started GET /queue/f1e044bedf7f37fb57361f115a62841b
>[negroni] Completed 200 OK in 142.23Âµs
>[negroni] Started GET /queue/84472ae559c45c9b84ab85d4a5852f91
>[negroni] Completed 200 OK in 218.451Âµs
>[negroni] Started GET /queue/f1e044bedf7f37fb57361f115a62841b
>[negroni] Completed 200 OK in 123.573Âµs
>[negroni] Started GET /queue/84472ae559c45c9b84ab85d4a5852f91
>[negroni] Completed 200 OK in 139.351Âµs
>[negroni] Started GET /queue/f1e044bedf7f37fb57361f115a62841b
>[negroni] Completed 200 OK in 141.994Âµs
>[negroni] Started GET /queue/84472ae559c45c9b84ab85d4a5852f91
>[negroni] Completed 200 OK in 138.308Âµs
>[negroni] Started GET /queue/f1e044bedf7f37fb57361f115a62841b
>[negroni] Completed 200 OK in 193.055Âµs
>[kubeexec] DEBUG 2018/06/08 08:42:32 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script volume stop vol_7851878e4341c93e517bff5cad8f7b49 force
>Result: volume stop: vol_7851878e4341c93e517bff5cad8f7b49: success
>[kubeexec] DEBUG 2018/06/08 08:42:32 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script volume delete vol_48f9db6f6531b638a436926286f6eefb
>Result: volume delete: vol_48f9db6f6531b638a436926286f6eefb: success
>[heketi] INFO 2018/06/08 08:42:32 Deleting brick 4c77a98309b18630c0618c89c97ba0e0
>[heketi] INFO 2018/06/08 08:42:32 Deleting brick 5b2e140dcb360bbba091b9c24633aec7
>[heketi] INFO 2018/06/08 08:42:32 Deleting brick dc9c085778d516278c0cadca4db4be15
>[kubeexec] DEBUG 2018/06/08 08:42:32 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mount | grep -w /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_dc9c085778d516278c0cadca4db4be15 | cut -d" " -f1
>Result: /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_dc9c085778d516278c0cadca4db4be15
>[kubeexec] DEBUG 2018/06/08 08:42:32 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mount | grep -w /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_4c77a98309b18630c0618c89c97ba0e0 | cut -d" " -f1
>Result: /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_4c77a98309b18630c0618c89c97ba0e0
>[kubeexec] DEBUG 2018/06/08 08:42:32 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script volume delete vol_7851878e4341c93e517bff5cad8f7b49
>Result: volume delete: vol_7851878e4341c93e517bff5cad8f7b49: success
>[heketi] INFO 2018/06/08 08:42:32 Deleting brick bfb72d4e8e105df407a890930c475175
>[heketi] INFO 2018/06/08 08:42:32 Deleting brick d7770ed15229b256bf58c081ad57ea73
>[heketi] INFO 2018/06/08 08:42:32 Deleting brick e5ae8fc12a741a851b2aade547fe3615
>[kubeexec] DEBUG 2018/06/08 08:42:32 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_4c77a98309b18630c0618c89c97ba0e0
>
>Result:   vg_9394bc70699b006c5460c9f654cf345f/tp_4c77a98309b18630c0618c89c97ba0e0
>[kubeexec] DEBUG 2018/06/08 08:42:32 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_dc9c085778d516278c0cadca4db4be15
>
>Result:   vg_d389f0278a774bd7443a09af960961d8/tp_dc9c085778d516278c0cadca4db4be15
>[negroni] Started GET /queue/84472ae559c45c9b84ab85d4a5852f91
>[negroni] Completed 200 OK in 114.315Âµs
>[kubeexec] DEBUG 2018/06/08 08:42:32 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mount | grep -w /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_5b2e140dcb360bbba091b9c24633aec7 | cut -d" " -f1
>Result: /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_5b2e140dcb360bbba091b9c24633aec7
>[negroni] Started GET /queue/f1e044bedf7f37fb57361f115a62841b
>[negroni] Completed 200 OK in 161.809Âµs
>[kubeexec] DEBUG 2018/06/08 08:42:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mount | grep -w /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_e5ae8fc12a741a851b2aade547fe3615 | cut -d" " -f1
>Result: /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_e5ae8fc12a741a851b2aade547fe3615
>[negroni] Started GET /queue/84472ae559c45c9b84ab85d4a5852f91
>[negroni] Completed 200 OK in 145.816Âµs
>[kubeexec] DEBUG 2018/06/08 08:42:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mount | grep -w /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_d7770ed15229b256bf58c081ad57ea73 | cut -d" " -f1
>Result: /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_d7770ed15229b256bf58c081ad57ea73
>[negroni] Started GET /queue/f1e044bedf7f37fb57361f115a62841b
>[negroni] Completed 200 OK in 171.342Âµs
>[kubeexec] DEBUG 2018/06/08 08:42:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mount | grep -w /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_bfb72d4e8e105df407a890930c475175 | cut -d" " -f1
>Result: /dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_bfb72d4e8e105df407a890930c475175
>[negroni] Started GET /queue/84472ae559c45c9b84ab85d4a5852f91
>[negroni] Completed 200 OK in 108.678Âµs
>[kubeexec] DEBUG 2018/06/08 08:42:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: umount /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_4c77a98309b18630c0618c89c97ba0e0
>Result: 
>[negroni] Started GET /queue/f1e044bedf7f37fb57361f115a62841b
>[negroni] Completed 200 OK in 123.278Âµs
>[kubeexec] DEBUG 2018/06/08 08:42:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: umount /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_dc9c085778d516278c0cadca4db4be15
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:42:35 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_5b2e140dcb360bbba091b9c24633aec7
>
>Result:   vg_3a4297677881963e3f80124971d50eea/tp_5b2e140dcb360bbba091b9c24633aec7
>[negroni] Started GET /queue/84472ae559c45c9b84ab85d4a5852f91
>[negroni] Completed 200 OK in 173.072Âµs
>[negroni] Started GET /queue/f1e044bedf7f37fb57361f115a62841b
>[negroni] Completed 200 OK in 111.529Âµs
>[kubeexec] DEBUG 2018/06/08 08:42:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_e5ae8fc12a741a851b2aade547fe3615
>
>Result:   vg_96f1667f2f1ced2c5ef94772922be93b/tp_e5ae8fc12a741a851b2aade547fe3615
>[kubeexec] DEBUG 2018/06/08 08:42:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_d7770ed15229b256bf58c081ad57ea73
>
>Result:   vg_d389f0278a774bd7443a09af960961d8/tp_d7770ed15229b256bf58c081ad57ea73
>[kubeexec] DEBUG 2018/06/08 08:42:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_bfb72d4e8e105df407a890930c475175
>
>Result:   vg_66a9af9f6bac95f9d8d556a2f14c29d3/tp_bfb72d4e8e105df407a890930c475175
>[negroni] Started GET /queue/84472ae559c45c9b84ab85d4a5852f91
>[negroni] Completed 200 OK in 130.875Âµs
>[negroni] Started GET /queue/f1e044bedf7f37fb57361f115a62841b
>[negroni] Completed 200 OK in 190.245Âµs
>[kubeexec] DEBUG 2018/06/08 08:42:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: sed -i.save "/brick_4c77a98309b18630c0618c89c97ba0e0/d" /var/lib/heketi/fstab
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:42:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: sed -i.save "/brick_dc9c085778d516278c0cadca4db4be15/d" /var/lib/heketi/fstab
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:42:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: umount /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_5b2e140dcb360bbba091b9c24633aec7
>Result: 
>[negroni] Started GET /queue/84472ae559c45c9b84ab85d4a5852f91
>[negroni] Completed 200 OK in 164.855Âµs
>[negroni] Started GET /queue/f1e044bedf7f37fb57361f115a62841b
>[negroni] Completed 200 OK in 118.761Âµs
>[kubeexec] DEBUG 2018/06/08 08:42:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: umount /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_e5ae8fc12a741a851b2aade547fe3615
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:42:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: umount /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_d7770ed15229b256bf58c081ad57ea73
>Result: 
>[negroni] Started GET /queue/84472ae559c45c9b84ab85d4a5852f91
>[negroni] Completed 200 OK in 159.016Âµs
>[kubeexec] DEBUG 2018/06/08 08:42:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: umount /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_bfb72d4e8e105df407a890930c475175
>Result: 
>[negroni] Started GET /queue/f1e044bedf7f37fb57361f115a62841b
>[negroni] Completed 200 OK in 163.439Âµs
>[kubeexec] DEBUG 2018/06/08 08:42:39 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvremove --autobackup=n -f /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_4c77a98309b18630c0618c89c97ba0e0
>
>Result:   Logical volume "brick_4c77a98309b18630c0618c89c97ba0e0" successfully removed
>[negroni] Started GET /queue/84472ae559c45c9b84ab85d4a5852f91
>[negroni] Completed 200 OK in 126.503Âµs
>[kubeexec] DEBUG 2018/06/08 08:42:39 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvremove --autobackup=n -f /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_dc9c085778d516278c0cadca4db4be15
>
>Result:   Logical volume "brick_dc9c085778d516278c0cadca4db4be15" successfully removed
>[negroni] Started GET /queue/f1e044bedf7f37fb57361f115a62841b
>[negroni] Completed 200 OK in 108.059Âµs
>[kubeexec] DEBUG 2018/06/08 08:42:39 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: sed -i.save "/brick_5b2e140dcb360bbba091b9c24633aec7/d" /var/lib/heketi/fstab
>Result: 
>[negroni] Started GET /queue/84472ae559c45c9b84ab85d4a5852f91
>[negroni] Completed 200 OK in 125.235Âµs
>[kubeexec] DEBUG 2018/06/08 08:42:40 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: sed -i.save "/brick_e5ae8fc12a741a851b2aade547fe3615/d" /var/lib/heketi/fstab
>Result: 
>[negroni] Started GET /queue/f1e044bedf7f37fb57361f115a62841b
>[negroni] Completed 200 OK in 106.963Âµs
>[kubeexec] DEBUG 2018/06/08 08:42:40 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: sed -i.save "/brick_d7770ed15229b256bf58c081ad57ea73/d" /var/lib/heketi/fstab
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:42:41 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: sed -i.save "/brick_bfb72d4e8e105df407a890930c475175/d" /var/lib/heketi/fstab
>Result: 
>[negroni] Started GET /queue/84472ae559c45c9b84ab85d4a5852f91
>[negroni] Completed 200 OK in 220.133Âµs
>[negroni] Started GET /queue/f1e044bedf7f37fb57361f115a62841b
>[negroni] Completed 200 OK in 171.973Âµs
>[kubeexec] DEBUG 2018/06/08 08:42:42 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvs --noheadings --options=thin_count   vg_9394bc70699b006c5460c9f654cf345f/tp_4c77a98309b18630c0618c89c97ba0e0
>
>Result:        0
>[kubeexec] DEBUG 2018/06/08 08:42:42 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvs --noheadings --options=thin_count   vg_d389f0278a774bd7443a09af960961d8/tp_dc9c085778d516278c0cadca4db4be15
>
>Result:        0
>[kubeexec] DEBUG 2018/06/08 08:42:42 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvremove --autobackup=n -f /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_5b2e140dcb360bbba091b9c24633aec7
>
>Result:   Logical volume "brick_5b2e140dcb360bbba091b9c24633aec7" successfully removed
>[negroni] Started GET /queue/84472ae559c45c9b84ab85d4a5852f91
>[negroni] Completed 200 OK in 110.57Âµs
>[negroni] Started GET /queue/f1e044bedf7f37fb57361f115a62841b
>[negroni] Completed 200 OK in 108.378Âµs
>[kubeexec] DEBUG 2018/06/08 08:42:43 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvremove --autobackup=n -f /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_e5ae8fc12a741a851b2aade547fe3615
>
>Result:   Logical volume "brick_e5ae8fc12a741a851b2aade547fe3615" successfully removed
>[kubeexec] DEBUG 2018/06/08 08:42:43 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvremove --autobackup=n -f /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_d7770ed15229b256bf58c081ad57ea73
>
>Result:   Logical volume "brick_d7770ed15229b256bf58c081ad57ea73" successfully removed
>[kubeexec] DEBUG 2018/06/08 08:42:43 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvremove --autobackup=n -f /dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_bfb72d4e8e105df407a890930c475175
>
>Result:   Logical volume "brick_bfb72d4e8e105df407a890930c475175" successfully removed
>[negroni] Started GET /queue/84472ae559c45c9b84ab85d4a5852f91
>[negroni] Completed 200 OK in 208.668Âµs
>[negroni] Started GET /queue/f1e044bedf7f37fb57361f115a62841b
>[negroni] Completed 200 OK in 127.372Âµs
>[kubeexec] DEBUG 2018/06/08 08:42:44 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvremove --autobackup=n -f   vg_9394bc70699b006c5460c9f654cf345f/tp_4c77a98309b18630c0618c89c97ba0e0
>
>Result:   Logical volume "tp_4c77a98309b18630c0618c89c97ba0e0" successfully removed
>[kubeexec] DEBUG 2018/06/08 08:42:44 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvremove --autobackup=n -f   vg_d389f0278a774bd7443a09af960961d8/tp_dc9c085778d516278c0cadca4db4be15
>
>Result:   Logical volume "tp_dc9c085778d516278c0cadca4db4be15" successfully removed
>[negroni] Started GET /queue/84472ae559c45c9b84ab85d4a5852f91
>[negroni] Completed 200 OK in 192.448Âµs
>[kubeexec] DEBUG 2018/06/08 08:42:44 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvs --noheadings --options=thin_count   vg_3a4297677881963e3f80124971d50eea/tp_5b2e140dcb360bbba091b9c24633aec7
>
>Result:        0
>[negroni] Started GET /queue/f1e044bedf7f37fb57361f115a62841b
>[negroni] Completed 200 OK in 164.198Âµs
>[kubeexec] DEBUG 2018/06/08 08:42:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvs --noheadings --options=thin_count   vg_96f1667f2f1ced2c5ef94772922be93b/tp_e5ae8fc12a741a851b2aade547fe3615
>
>Result:        0
>[negroni] Started GET /queue/84472ae559c45c9b84ab85d4a5852f91
>[negroni] Completed 200 OK in 174.553Âµs
>[kubeexec] DEBUG 2018/06/08 08:42:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvs --noheadings --options=thin_count   vg_d389f0278a774bd7443a09af960961d8/tp_d7770ed15229b256bf58c081ad57ea73
>
>Result:        0
>[negroni] Started GET /queue/f1e044bedf7f37fb57361f115a62841b
>[negroni] Completed 200 OK in 116.448Âµs
>[kubeexec] DEBUG 2018/06/08 08:42:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvs --noheadings --options=thin_count   vg_66a9af9f6bac95f9d8d556a2f14c29d3/tp_bfb72d4e8e105df407a890930c475175
>
>Result:        0
>[negroni] Started GET /queue/84472ae559c45c9b84ab85d4a5852f91
>[negroni] Completed 200 OK in 116.179Âµs
>[kubeexec] DEBUG 2018/06/08 08:42:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: rmdir /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_4c77a98309b18630c0618c89c97ba0e0
>Result: 
>[negroni] Started GET /queue/f1e044bedf7f37fb57361f115a62841b
>[negroni] Completed 200 OK in 111.506Âµs
>[kubeexec] DEBUG 2018/06/08 08:42:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: rmdir /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_dc9c085778d516278c0cadca4db4be15
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:42:47 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvremove --autobackup=n -f   vg_3a4297677881963e3f80124971d50eea/tp_5b2e140dcb360bbba091b9c24633aec7
>
>Result:   Logical volume "tp_5b2e140dcb360bbba091b9c24633aec7" successfully removed
>[negroni] Started GET /queue/84472ae559c45c9b84ab85d4a5852f91
>[negroni] Completed 200 OK in 162.504Âµs
>[negroni] Started GET /queue/f1e044bedf7f37fb57361f115a62841b
>[negroni] Completed 200 OK in 182.842Âµs
>[kubeexec] DEBUG 2018/06/08 08:42:48 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvremove --autobackup=n -f   vg_96f1667f2f1ced2c5ef94772922be93b/tp_e5ae8fc12a741a851b2aade547fe3615
>
>Result:   Logical volume "tp_e5ae8fc12a741a851b2aade547fe3615" successfully removed
>[kubeexec] DEBUG 2018/06/08 08:42:48 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvremove --autobackup=n -f   vg_d389f0278a774bd7443a09af960961d8/tp_d7770ed15229b256bf58c081ad57ea73
>
>Result:   Logical volume "tp_d7770ed15229b256bf58c081ad57ea73" successfully removed
>[kubeexec] DEBUG 2018/06/08 08:42:48 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvremove --autobackup=n -f   vg_66a9af9f6bac95f9d8d556a2f14c29d3/tp_bfb72d4e8e105df407a890930c475175
>
>Result:   Logical volume "tp_bfb72d4e8e105df407a890930c475175" successfully removed
>[negroni] Started GET /queue/84472ae559c45c9b84ab85d4a5852f91
>[negroni] Completed 200 OK in 172.232Âµs
>[negroni] Started GET /queue/f1e044bedf7f37fb57361f115a62841b
>[negroni] Completed 200 OK in 175.795Âµs
>[kubeexec] DEBUG 2018/06/08 08:42:49 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: rmdir /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_e5ae8fc12a741a851b2aade547fe3615
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:42:49 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: rmdir /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_d7770ed15229b256bf58c081ad57ea73
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:42:49 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: rmdir /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_5b2e140dcb360bbba091b9c24633aec7
>Result: 
>[heketi] INFO 2018/06/08 08:42:49 Delete Volume succeeded
>[asynchttp] INFO 2018/06/08 08:42:49 asynchttp.go:292: Completed job 84472ae559c45c9b84ab85d4a5852f91 in 23.865479238s
>[negroni] Started GET /queue/84472ae559c45c9b84ab85d4a5852f91
>[negroni] Completed 204 No Content in 164.556Âµs
>[negroni] Started GET /queue/f1e044bedf7f37fb57361f115a62841b
>[negroni] Completed 200 OK in 106.769Âµs
>[kubeexec] DEBUG 2018/06/08 08:42:49 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: rmdir /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_bfb72d4e8e105df407a890930c475175
>Result: 
>[heketi] INFO 2018/06/08 08:42:50 Delete Volume succeeded
>[asynchttp] INFO 2018/06/08 08:42:50 asynchttp.go:292: Completed job f1e044bedf7f37fb57361f115a62841b in 24.052834729s
>[negroni] Started GET /queue/f1e044bedf7f37fb57361f115a62841b
>[negroni] Completed 204 No Content in 157.203Âµs
>[negroni] Started DELETE /volumes/7851878e4341c93e517bff5cad8f7b49
>[negroni] Completed 404 Not Found in 3.675742ms
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 176.799Âµs
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 3.00093ms
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 1.158224ms
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 1.668917ms
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 184.905Âµs
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 705.014Âµs
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 1.189942ms
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 1.088299ms
>[negroni] Started POST /volumes
>[heketi] INFO 2018/06/08 08:42:52 Allocating brick set #0
>[negroni] Completed 202 Accepted in 13.881185ms
>[asynchttp] INFO 2018/06/08 08:42:52 asynchttp.go:288: Started job 44168b106c812de9f08e3a8754d679af
>[heketi] INFO 2018/06/08 08:42:52 Started async operation: Create Volume
>[negroni] Started GET /queue/44168b106c812de9f08e3a8754d679af
>[negroni] Completed 200 OK in 88.113Âµs
>[heketi] INFO 2018/06/08 08:42:52 Creating brick a48060da658afcf0bd672a1d5e325a35
>[heketi] INFO 2018/06/08 08:42:52 Creating brick eab93f37cd264f7b8d5185da3d5e1ff2
>[heketi] INFO 2018/06/08 08:42:52 Creating brick 6041796df9e6e8dde7acbc28fb15eebd
>[negroni] Started POST /volumes
>[heketi] INFO 2018/06/08 08:42:52 Allocating brick set #0
>[negroni] Completed 202 Accepted in 17.146336ms
>[asynchttp] INFO 2018/06/08 08:42:52 asynchttp.go:288: Started job f4bd066349a28138ceafa752530e031d
>[heketi] INFO 2018/06/08 08:42:52 Started async operation: Create Volume
>[negroni] Started GET /queue/f4bd066349a28138ceafa752530e031d
>[negroni] Completed 200 OK in 110.896Âµs
>[heketi] INFO 2018/06/08 08:42:52 Creating brick 9b2ad28a2bba6499ea37c90a18a2f09e
>[heketi] INFO 2018/06/08 08:42:52 Creating brick 92deb78ca12e07cbb3500736c89f75fb
>[heketi] INFO 2018/06/08 08:42:52 Creating brick 69c43b1a8d9e0514a7e21bde6a2b8a94
>[kubeexec] DEBUG 2018/06/08 08:42:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir -p /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_a48060da658afcf0bd672a1d5e325a35
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:42:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir -p /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_6041796df9e6e8dde7acbc28fb15eebd
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:42:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir -p /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_eab93f37cd264f7b8d5185da3d5e1ff2
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:42:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_d389f0278a774bd7443a09af960961d8/tp_eab93f37cd264f7b8d5185da3d5e1ff2 --virtualsize 2097152K --name brick_eab93f37cd264f7b8d5185da3d5e1ff2
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_eab93f37cd264f7b8d5185da3d5e1ff2" created.
>[kubeexec] DEBUG 2018/06/08 08:42:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_9394bc70699b006c5460c9f654cf345f/tp_a48060da658afcf0bd672a1d5e325a35 --virtualsize 2097152K --name brick_a48060da658afcf0bd672a1d5e325a35
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_a48060da658afcf0bd672a1d5e325a35" created.
>[kubeexec] DEBUG 2018/06/08 08:42:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_66a9af9f6bac95f9d8d556a2f14c29d3/tp_6041796df9e6e8dde7acbc28fb15eebd --virtualsize 2097152K --name brick_6041796df9e6e8dde7acbc28fb15eebd
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_6041796df9e6e8dde7acbc28fb15eebd" created.
>[kubeexec] DEBUG 2018/06/08 08:42:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_a48060da658afcf0bd672a1d5e325a35
>Result: meta-data=/dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_a48060da658afcf0bd672a1d5e325a35 isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:42:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_6041796df9e6e8dde7acbc28fb15eebd
>Result: meta-data=/dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_6041796df9e6e8dde7acbc28fb15eebd isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:42:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_eab93f37cd264f7b8d5185da3d5e1ff2
>Result: meta-data=/dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_eab93f37cd264f7b8d5185da3d5e1ff2 isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:42:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: awk "BEGIN {print \"/dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_a48060da658afcf0bd672a1d5e325a35 /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_a48060da658afcf0bd672a1d5e325a35 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:42:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: awk "BEGIN {print \"/dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_6041796df9e6e8dde7acbc28fb15eebd /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_6041796df9e6e8dde7acbc28fb15eebd xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:42:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: awk "BEGIN {print \"/dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_eab93f37cd264f7b8d5185da3d5e1ff2 /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_eab93f37cd264f7b8d5185da3d5e1ff2 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:42:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_eab93f37cd264f7b8d5185da3d5e1ff2 /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_eab93f37cd264f7b8d5185da3d5e1ff2
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:42:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_a48060da658afcf0bd672a1d5e325a35 /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_a48060da658afcf0bd672a1d5e325a35
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:42:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_6041796df9e6e8dde7acbc28fb15eebd /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_6041796df9e6e8dde7acbc28fb15eebd
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:42:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_eab93f37cd264f7b8d5185da3d5e1ff2/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:42:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_a48060da658afcf0bd672a1d5e325a35/brick
>Result: 
>[negroni] Started GET /queue/44168b106c812de9f08e3a8754d679af
>[negroni] Completed 200 OK in 140.028Âµs
>[kubeexec] DEBUG 2018/06/08 08:42:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_6041796df9e6e8dde7acbc28fb15eebd/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:42:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: chown :2000 /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_a48060da658afcf0bd672a1d5e325a35/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:42:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: chown :2000 /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_eab93f37cd264f7b8d5185da3d5e1ff2/brick
>Result: 
>[negroni] Started GET /queue/f4bd066349a28138ceafa752530e031d
>[negroni] Completed 200 OK in 93.878Âµs
>[kubeexec] DEBUG 2018/06/08 08:42:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: chown :2000 /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_6041796df9e6e8dde7acbc28fb15eebd/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:42:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: chmod 2775 /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_a48060da658afcf0bd672a1d5e325a35/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:42:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: chmod 2775 /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_eab93f37cd264f7b8d5185da3d5e1ff2/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:42:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: chmod 2775 /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_6041796df9e6e8dde7acbc28fb15eebd/brick
>Result: 
>[cmdexec] INFO 2018/06/08 08:42:53 Creating volume vol_519804da0a04715f5fe88eef3554e109 replica 3
>[kubeexec] DEBUG 2018/06/08 08:42:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir -p /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_69c43b1a8d9e0514a7e21bde6a2b8a94
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:42:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir -p /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_92deb78ca12e07cbb3500736c89f75fb
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:42:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir -p /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_9b2ad28a2bba6499ea37c90a18a2f09e
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:42:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_d389f0278a774bd7443a09af960961d8/tp_69c43b1a8d9e0514a7e21bde6a2b8a94 --virtualsize 2097152K --name brick_69c43b1a8d9e0514a7e21bde6a2b8a94
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_69c43b1a8d9e0514a7e21bde6a2b8a94" created.
>[kubeexec] DEBUG 2018/06/08 08:42:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_3a4297677881963e3f80124971d50eea/tp_92deb78ca12e07cbb3500736c89f75fb --virtualsize 2097152K --name brick_92deb78ca12e07cbb3500736c89f75fb
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_92deb78ca12e07cbb3500736c89f75fb" created.
>[kubeexec] DEBUG 2018/06/08 08:42:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_9394bc70699b006c5460c9f654cf345f/tp_9b2ad28a2bba6499ea37c90a18a2f09e --virtualsize 2097152K --name brick_9b2ad28a2bba6499ea37c90a18a2f09e
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_9b2ad28a2bba6499ea37c90a18a2f09e" created.
>[kubeexec] DEBUG 2018/06/08 08:42:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_69c43b1a8d9e0514a7e21bde6a2b8a94
>Result: meta-data=/dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_69c43b1a8d9e0514a7e21bde6a2b8a94 isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:42:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_92deb78ca12e07cbb3500736c89f75fb
>Result: meta-data=/dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_92deb78ca12e07cbb3500736c89f75fb isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:42:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_9b2ad28a2bba6499ea37c90a18a2f09e
>Result: meta-data=/dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_9b2ad28a2bba6499ea37c90a18a2f09e isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:42:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: awk "BEGIN {print \"/dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_92deb78ca12e07cbb3500736c89f75fb /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_92deb78ca12e07cbb3500736c89f75fb xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:42:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: awk "BEGIN {print \"/dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_69c43b1a8d9e0514a7e21bde6a2b8a94 /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_69c43b1a8d9e0514a7e21bde6a2b8a94 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:42:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: awk "BEGIN {print \"/dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_9b2ad28a2bba6499ea37c90a18a2f09e /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_9b2ad28a2bba6499ea37c90a18a2f09e xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:42:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_92deb78ca12e07cbb3500736c89f75fb /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_92deb78ca12e07cbb3500736c89f75fb
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:42:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_69c43b1a8d9e0514a7e21bde6a2b8a94 /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_69c43b1a8d9e0514a7e21bde6a2b8a94
>Result: 
>[negroni] Started GET /queue/44168b106c812de9f08e3a8754d679af
>[negroni] Completed 200 OK in 104.967Âµs
>[kubeexec] DEBUG 2018/06/08 08:42:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_9b2ad28a2bba6499ea37c90a18a2f09e /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_9b2ad28a2bba6499ea37c90a18a2f09e
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:42:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_92deb78ca12e07cbb3500736c89f75fb/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:42:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_69c43b1a8d9e0514a7e21bde6a2b8a94/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:42:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_9b2ad28a2bba6499ea37c90a18a2f09e/brick
>Result: 
>[negroni] Started GET /queue/f4bd066349a28138ceafa752530e031d
>[negroni] Completed 200 OK in 85.8Âµs
>[kubeexec] DEBUG 2018/06/08 08:42:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: chown :2001 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_92deb78ca12e07cbb3500736c89f75fb/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:42:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: chown :2001 /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_69c43b1a8d9e0514a7e21bde6a2b8a94/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:42:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: chown :2001 /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_9b2ad28a2bba6499ea37c90a18a2f09e/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:42:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: chmod 2775 /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_69c43b1a8d9e0514a7e21bde6a2b8a94/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:42:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: chmod 2775 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_92deb78ca12e07cbb3500736c89f75fb/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:42:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: chmod 2775 /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_9b2ad28a2bba6499ea37c90a18a2f09e/brick
>Result: 
>[cmdexec] INFO 2018/06/08 08:42:54 Creating volume vol_34180b600b060c8400f7902d50d19fce replica 3
>[kubeexec] DEBUG 2018/06/08 08:42:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script volume create vol_34180b600b060c8400f7902d50d19fce replica 3 10.70.46.122:/var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_92deb78ca12e07cbb3500736c89f75fb/brick 10.70.47.76:/var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_9b2ad28a2bba6499ea37c90a18a2f09e/brick 10.70.46.187:/var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_69c43b1a8d9e0514a7e21bde6a2b8a94/brick
>Result: volume create: vol_34180b600b060c8400f7902d50d19fce: success: please start the volume to access data
>[kubeexec] DEBUG 2018/06/08 08:42:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script volume create vol_519804da0a04715f5fe88eef3554e109 replica 3 10.70.47.76:/var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_a48060da658afcf0bd672a1d5e325a35/brick 10.70.46.187:/var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_eab93f37cd264f7b8d5185da3d5e1ff2/brick 10.70.46.122:/var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_6041796df9e6e8dde7acbc28fb15eebd/brick
>Result: volume create: vol_519804da0a04715f5fe88eef3554e109: success: please start the volume to access data
>[negroni] Started GET /queue/44168b106c812de9f08e3a8754d679af
>[negroni] Completed 200 OK in 338.092Âµs
>[negroni] Started GET /queue/f4bd066349a28138ceafa752530e031d
>[negroni] Completed 200 OK in 150.507Âµs
>[negroni] Started GET /queue/44168b106c812de9f08e3a8754d679af
>[negroni] Completed 200 OK in 109.564Âµs
>[negroni] Started GET /queue/f4bd066349a28138ceafa752530e031d
>[negroni] Completed 200 OK in 116.247Âµs
>[negroni] Started GET /queue/44168b106c812de9f08e3a8754d679af
>[negroni] Completed 200 OK in 121.895Âµs
>[negroni] Started GET /queue/f4bd066349a28138ceafa752530e031d
>[negroni] Completed 200 OK in 118.864Âµs
>[negroni] Started GET /queue/44168b106c812de9f08e3a8754d679af
>[negroni] Completed 200 OK in 109.534Âµs
>[negroni] Started GET /queue/f4bd066349a28138ceafa752530e031d
>[negroni] Completed 200 OK in 118.054Âµs
>[negroni] Started GET /queue/44168b106c812de9f08e3a8754d679af
>[negroni] Completed 200 OK in 171.078Âµs
>[negroni] Started GET /queue/f4bd066349a28138ceafa752530e031d
>[negroni] Completed 200 OK in 170.979Âµs
>[negroni] Started GET /queue/44168b106c812de9f08e3a8754d679af
>[negroni] Completed 200 OK in 188.273Âµs
>[negroni] Started GET /queue/f4bd066349a28138ceafa752530e031d
>[negroni] Completed 200 OK in 231.622Âµs
>[negroni] Started GET /queue/44168b106c812de9f08e3a8754d679af
>[negroni] Completed 200 OK in 144.627Âµs
>[negroni] Started GET /queue/f4bd066349a28138ceafa752530e031d
>[negroni] Completed 200 OK in 169.804Âµs
>[kubeexec] DEBUG 2018/06/08 08:43:01 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script volume start vol_34180b600b060c8400f7902d50d19fce
>Result: volume start: vol_34180b600b060c8400f7902d50d19fce: success
>[kubeexec] DEBUG 2018/06/08 08:43:01 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script volume start vol_519804da0a04715f5fe88eef3554e109
>Result: volume start: vol_519804da0a04715f5fe88eef3554e109: success
>[heketi] INFO 2018/06/08 08:43:02 Create Volume succeeded
>[asynchttp] INFO 2018/06/08 08:43:02 asynchttp.go:292: Completed job f4bd066349a28138ceafa752530e031d in 9.792582551s
>[heketi] INFO 2018/06/08 08:43:02 Create Volume succeeded
>[asynchttp] INFO 2018/06/08 08:43:02 asynchttp.go:292: Completed job 44168b106c812de9f08e3a8754d679af in 9.90405144s
>[negroni] Started GET /queue/44168b106c812de9f08e3a8754d679af
>[negroni] Completed 303 See Other in 206.983Âµs
>[negroni] Started GET /volumes/519804da0a04715f5fe88eef3554e109
>[negroni] Completed 200 OK in 7.06093ms
>[negroni] Started GET /clusters/0a73c60efdd4673113b668afea101e6d
>[negroni] Completed 200 OK in 201.816Âµs
>[negroni] Started GET /nodes/278bd6b4e16a8e62ef15aaae22e6abc1
>[negroni] Completed 200 OK in 4.061402ms
>[negroni] Started GET /nodes/70423cc0bcd044fe5ba8bbbd256a3e49
>[negroni] Completed 200 OK in 2.282748ms
>[negroni] Started GET /nodes/d942fe6c0ee5691b7cc263968f97b650
>[negroni] Completed 200 OK in 1.356823ms
>[negroni] Started GET /queue/f4bd066349a28138ceafa752530e031d
>[negroni] Completed 303 See Other in 181.552Âµs
>[negroni] Started GET /volumes/34180b600b060c8400f7902d50d19fce
>[negroni] Completed 200 OK in 784.081Âµs
>[negroni] Started GET /clusters/0a73c60efdd4673113b668afea101e6d
>[negroni] Completed 200 OK in 469.804Âµs
>[negroni] Started GET /nodes/278bd6b4e16a8e62ef15aaae22e6abc1
>[negroni] Completed 200 OK in 1.131032ms
>[negroni] Started GET /nodes/70423cc0bcd044fe5ba8bbbd256a3e49
>[negroni] Completed 200 OK in 794.968Âµs
>[negroni] Started GET /nodes/d942fe6c0ee5691b7cc263968f97b650
>[negroni] Completed 200 OK in 736.634Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 1.028392ms
>[negroni] Started GET /volumes/34180b600b060c8400f7902d50d19fce
>[negroni] Completed 200 OK in 1.130505ms
>[negroni] Started GET /volumes/519804da0a04715f5fe88eef3554e109
>[negroni] Completed 200 OK in 1.029772ms
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 623.296Âµs
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 999.173Âµs
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 1.75559ms
>[negroni] Started DELETE /volumes/34180b600b060c8400f7902d50d19fce
>[negroni] Completed 202 Accepted in 9.569326ms
>[asynchttp] INFO 2018/06/08 08:43:04 asynchttp.go:288: Started job 346c61e53782b5e0c258f3a62595da0d
>[heketi] INFO 2018/06/08 08:43:04 Started async operation: Delete Volume
>[negroni] Started GET /queue/346c61e53782b5e0c258f3a62595da0d
>[negroni] Completed 200 OK in 159.842Âµs
>[negroni] Started DELETE /volumes/519804da0a04715f5fe88eef3554e109
>[negroni] Completed 202 Accepted in 11.791578ms
>[asynchttp] INFO 2018/06/08 08:43:04 asynchttp.go:288: Started job 68f1753cbf11e3a7962681eb39bdfd38
>[heketi] INFO 2018/06/08 08:43:04 Started async operation: Delete Volume
>[negroni] Started GET /queue/68f1753cbf11e3a7962681eb39bdfd38
>[negroni] Completed 200 OK in 102.088Âµs
>[kubeexec] DEBUG 2018/06/08 08:43:04 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: gluster --mode=script snapshot list vol_34180b600b060c8400f7902d50d19fce --xml
>Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
><cliOutput>
>  <opRet>0</opRet>
>  <opErrno>0</opErrno>
>  <opErrstr/>
>  <snapList>
>    <count>0</count>
>  </snapList>
></cliOutput>
>[kubeexec] DEBUG 2018/06/08 08:43:04 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script snapshot list vol_519804da0a04715f5fe88eef3554e109 --xml
>Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
><cliOutput>
>  <opRet>0</opRet>
>  <opErrno>0</opErrno>
>  <opErrstr/>
>  <snapList>
>    <count>0</count>
>  </snapList>
></cliOutput>
>[negroni] Started GET /queue/346c61e53782b5e0c258f3a62595da0d
>[negroni] Completed 200 OK in 195.879Âµs
>[negroni] Started GET /queue/68f1753cbf11e3a7962681eb39bdfd38
>[negroni] Completed 200 OK in 128.573Âµs
>[negroni] Started GET /queue/346c61e53782b5e0c258f3a62595da0d
>[negroni] Completed 200 OK in 136.535Âµs
>[negroni] Started GET /queue/68f1753cbf11e3a7962681eb39bdfd38
>[negroni] Completed 200 OK in 165.739Âµs
>[kubeexec] DEBUG 2018/06/08 08:43:06 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: gluster --mode=script volume stop vol_34180b600b060c8400f7902d50d19fce force
>Result: volume stop: vol_34180b600b060c8400f7902d50d19fce: success
>[negroni] Started GET /queue/346c61e53782b5e0c258f3a62595da0d
>[negroni] Completed 200 OK in 142.766Âµs
>[negroni] Started GET /queue/68f1753cbf11e3a7962681eb39bdfd38
>[negroni] Completed 200 OK in 211.655Âµs
>[negroni] Started GET /queue/346c61e53782b5e0c258f3a62595da0d
>[negroni] Completed 200 OK in 245.952Âµs
>[negroni] Started GET /queue/68f1753cbf11e3a7962681eb39bdfd38
>[negroni] Completed 200 OK in 183.233Âµs
>[negroni] Started GET /queue/346c61e53782b5e0c258f3a62595da0d
>[negroni] Completed 200 OK in 176.933Âµs
>[negroni] Started GET /queue/68f1753cbf11e3a7962681eb39bdfd38
>[negroni] Completed 200 OK in 156.674Âµs
>[negroni] Started GET /queue/346c61e53782b5e0c258f3a62595da0d
>[negroni] Completed 200 OK in 124.654Âµs
>[negroni] Started GET /queue/68f1753cbf11e3a7962681eb39bdfd38
>[negroni] Completed 200 OK in 123.546Âµs
>[kubeexec] DEBUG 2018/06/08 08:43:10 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script volume stop vol_519804da0a04715f5fe88eef3554e109 force
>Result: volume stop: vol_519804da0a04715f5fe88eef3554e109: success
>[kubeexec] DEBUG 2018/06/08 08:43:10 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: gluster --mode=script volume delete vol_34180b600b060c8400f7902d50d19fce
>Result: volume delete: vol_34180b600b060c8400f7902d50d19fce: success
>[heketi] INFO 2018/06/08 08:43:10 Deleting brick 69c43b1a8d9e0514a7e21bde6a2b8a94
>[heketi] INFO 2018/06/08 08:43:10 Deleting brick 92deb78ca12e07cbb3500736c89f75fb
>[heketi] INFO 2018/06/08 08:43:10 Deleting brick 9b2ad28a2bba6499ea37c90a18a2f09e
>[kubeexec] DEBUG 2018/06/08 08:43:11 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mount | grep -w /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_69c43b1a8d9e0514a7e21bde6a2b8a94 | cut -d" " -f1
>Result: /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_69c43b1a8d9e0514a7e21bde6a2b8a94
>[kubeexec] DEBUG 2018/06/08 08:43:11 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mount | grep -w /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_9b2ad28a2bba6499ea37c90a18a2f09e | cut -d" " -f1
>Result: /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_9b2ad28a2bba6499ea37c90a18a2f09e
>[kubeexec] DEBUG 2018/06/08 08:43:11 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script volume delete vol_519804da0a04715f5fe88eef3554e109
>Result: volume delete: vol_519804da0a04715f5fe88eef3554e109: success
>[heketi] INFO 2018/06/08 08:43:11 Deleting brick a48060da658afcf0bd672a1d5e325a35
>[heketi] INFO 2018/06/08 08:43:11 Deleting brick eab93f37cd264f7b8d5185da3d5e1ff2
>[heketi] INFO 2018/06/08 08:43:11 Deleting brick 6041796df9e6e8dde7acbc28fb15eebd
>[kubeexec] DEBUG 2018/06/08 08:43:11 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_69c43b1a8d9e0514a7e21bde6a2b8a94
>
>Result:   vg_d389f0278a774bd7443a09af960961d8/tp_69c43b1a8d9e0514a7e21bde6a2b8a94
>[kubeexec] DEBUG 2018/06/08 08:43:11 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_9b2ad28a2bba6499ea37c90a18a2f09e
>
>Result:   vg_9394bc70699b006c5460c9f654cf345f/tp_9b2ad28a2bba6499ea37c90a18a2f09e
>[negroni] Started GET /queue/346c61e53782b5e0c258f3a62595da0d
>[negroni] Completed 200 OK in 197.323Âµs
>[kubeexec] DEBUG 2018/06/08 08:43:11 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mount | grep -w /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_92deb78ca12e07cbb3500736c89f75fb | cut -d" " -f1
>Result: /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_92deb78ca12e07cbb3500736c89f75fb
>[negroni] Started GET /queue/68f1753cbf11e3a7962681eb39bdfd38
>[negroni] Completed 200 OK in 217.161Âµs
>[kubeexec] DEBUG 2018/06/08 08:43:12 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mount | grep -w /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_eab93f37cd264f7b8d5185da3d5e1ff2 | cut -d" " -f1
>Result: /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_eab93f37cd264f7b8d5185da3d5e1ff2
>[kubeexec] DEBUG 2018/06/08 08:43:12 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mount | grep -w /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_a48060da658afcf0bd672a1d5e325a35 | cut -d" " -f1
>Result: /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_a48060da658afcf0bd672a1d5e325a35
>[negroni] Started GET /queue/346c61e53782b5e0c258f3a62595da0d
>[negroni] Completed 200 OK in 140.577Âµs
>[negroni] Started GET /queue/68f1753cbf11e3a7962681eb39bdfd38
>[negroni] Completed 200 OK in 188.552Âµs
>[kubeexec] DEBUG 2018/06/08 08:43:12 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mount | grep -w /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_6041796df9e6e8dde7acbc28fb15eebd | cut -d" " -f1
>Result: /dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_6041796df9e6e8dde7acbc28fb15eebd
>[negroni] Started GET /queue/346c61e53782b5e0c258f3a62595da0d
>[negroni] Completed 200 OK in 147.798Âµs
>[kubeexec] DEBUG 2018/06/08 08:43:13 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: umount /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_69c43b1a8d9e0514a7e21bde6a2b8a94
>Result: 
>[negroni] Started GET /queue/68f1753cbf11e3a7962681eb39bdfd38
>[negroni] Completed 200 OK in 110.875Âµs
>[kubeexec] DEBUG 2018/06/08 08:43:13 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: umount /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_9b2ad28a2bba6499ea37c90a18a2f09e
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:43:13 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_92deb78ca12e07cbb3500736c89f75fb
>
>Result:   vg_3a4297677881963e3f80124971d50eea/tp_92deb78ca12e07cbb3500736c89f75fb
>[heketi] INFO 2018/06/08 08:43:14 Starting Node Health Status refresh
>[cmdexec] INFO 2018/06/08 08:43:14 Check Glusterd service status in node dhcp46-187.lab.eng.blr.redhat.com
>[negroni] Started GET /queue/346c61e53782b5e0c258f3a62595da0d
>[negroni] Completed 200 OK in 189.432Âµs
>[negroni] Started GET /queue/68f1753cbf11e3a7962681eb39bdfd38
>[negroni] Completed 200 OK in 119.201Âµs
>[kubeexec] DEBUG 2018/06/08 08:43:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_eab93f37cd264f7b8d5185da3d5e1ff2
>
>Result:   vg_d389f0278a774bd7443a09af960961d8/tp_eab93f37cd264f7b8d5185da3d5e1ff2
>[kubeexec] DEBUG 2018/06/08 08:43:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_a48060da658afcf0bd672a1d5e325a35
>
>Result:   vg_9394bc70699b006c5460c9f654cf345f/tp_a48060da658afcf0bd672a1d5e325a35
>[kubeexec] DEBUG 2018/06/08 08:43:15 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_6041796df9e6e8dde7acbc28fb15eebd
>
>Result:   vg_66a9af9f6bac95f9d8d556a2f14c29d3/tp_6041796df9e6e8dde7acbc28fb15eebd
>[negroni] Started GET /queue/346c61e53782b5e0c258f3a62595da0d
>[negroni] Completed 200 OK in 247.618Âµs
>[negroni] Started GET /queue/68f1753cbf11e3a7962681eb39bdfd38
>[negroni] Completed 200 OK in 139.058Âµs
>[kubeexec] DEBUG 2018/06/08 08:43:15 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: sed -i.save "/brick_69c43b1a8d9e0514a7e21bde6a2b8a94/d" /var/lib/heketi/fstab
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:43:15 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: sed -i.save "/brick_9b2ad28a2bba6499ea37c90a18a2f09e/d" /var/lib/heketi/fstab
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:43:16 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: umount /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_92deb78ca12e07cbb3500736c89f75fb
>Result: 
>[negroni] Started GET /queue/346c61e53782b5e0c258f3a62595da0d
>[negroni] Completed 200 OK in 127.792Âµs
>[negroni] Started GET /queue/68f1753cbf11e3a7962681eb39bdfd38
>[negroni] Completed 200 OK in 191.007Âµs
>[kubeexec] DEBUG 2018/06/08 08:43:16 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:33 UTC; 2h 19min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48fed567_6ae4_11e8_bc4b_005056a5f18a.slice/docker-a506d9aaca4b440d184df40d01e90b2040811c8090bf4c19d892081307b28c0c.scope/system.slice/glusterd.service
>           ââ  433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1228 /usr/sbin/glusterfsd -s 10.70.46.187 --volfile-id heketidbstorage.10.70.46.187.var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.187-var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.pid -S /var/run/gluster/c08898c2e3b0190c62df3a9d4a013c92.socket --brick-name /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.log --xlator-option *-posix.glusterd-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ13429 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/8fb729a0bbd51b204491c603c84acaa4.socket --xlator-option *replicate*.node-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034
>
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Unit glusterd.service entered failed state.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: glusterd.service failed.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:33 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 08:43:16 Periodic health check status: node 278bd6b4e16a8e62ef15aaae22e6abc1 up=true
>[cmdexec] INFO 2018/06/08 08:43:16 Check Glusterd service status in node dhcp46-122.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 08:43:17 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: umount /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_a48060da658afcf0bd672a1d5e325a35
>Result: 
>[negroni] Started GET /queue/346c61e53782b5e0c258f3a62595da0d
>[negroni] Completed 200 OK in 120.942Âµs
>[kubeexec] DEBUG 2018/06/08 08:43:17 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: umount /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_6041796df9e6e8dde7acbc28fb15eebd
>Result: 
>[negroni] Started GET /queue/68f1753cbf11e3a7962681eb39bdfd38
>[negroni] Completed 200 OK in 240.088Âµs
>[kubeexec] DEBUG 2018/06/08 08:43:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: umount /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_eab93f37cd264f7b8d5185da3d5e1ff2
>Result: 
>[negroni] Started GET /queue/346c61e53782b5e0c258f3a62595da0d
>[negroni] Completed 200 OK in 119.052Âµs
>[kubeexec] DEBUG 2018/06/08 08:43:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvremove --autobackup=n -f /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_9b2ad28a2bba6499ea37c90a18a2f09e
>
>Result:   Logical volume "brick_9b2ad28a2bba6499ea37c90a18a2f09e" successfully removed
>[negroni] Started GET /queue/68f1753cbf11e3a7962681eb39bdfd38
>[negroni] Completed 200 OK in 124.418Âµs
>[kubeexec] DEBUG 2018/06/08 08:43:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: sed -i.save "/brick_92deb78ca12e07cbb3500736c89f75fb/d" /var/lib/heketi/fstab
>Result: 
>[negroni] Started GET /queue/346c61e53782b5e0c258f3a62595da0d
>[negroni] Completed 200 OK in 129.859Âµs
>[kubeexec] DEBUG 2018/06/08 08:43:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvremove --autobackup=n -f /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_69c43b1a8d9e0514a7e21bde6a2b8a94
>
>Result:   Logical volume "brick_69c43b1a8d9e0514a7e21bde6a2b8a94" successfully removed
>[negroni] Started GET /queue/68f1753cbf11e3a7962681eb39bdfd38
>[negroni] Completed 200 OK in 120.116Âµs
>[kubeexec] DEBUG 2018/06/08 08:43:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: sed -i.save "/brick_a48060da658afcf0bd672a1d5e325a35/d" /var/lib/heketi/fstab
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:43:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:42 UTC; 2h 19min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4900c968_6ae4_11e8_bc4b_005056a5f18a.slice/docker-438fb5fb16a82a3e701b28fa1f1ef8623dc37ca06374c604e5781a2b2ab2e06d.scope/system.slice/glusterd.service
>           ââ  433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1106 /usr/sbin/glusterfsd -s 10.70.46.122 --volfile-id heketidbstorage.10.70.46.122.var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.122-var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.pid -S /var/run/gluster/9fa3ac3b12826912287adc88c26ac134.socket --brick-name /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_29e5b36d921cca1b0699871d9727c932/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.log --xlator-option *-posix.glusterd-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ13712 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f8994cdb09351d83e70da08261d99c64.socket --xlator-option *replicate*.node-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0
>
>Jun 08 06:23:40 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:42 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 08:43:19 Periodic health check status: node 70423cc0bcd044fe5ba8bbbd256a3e49 up=true
>[cmdexec] INFO 2018/06/08 08:43:19 Check Glusterd service status in node dhcp47-76.lab.eng.blr.redhat.com
>[negroni] Started GET /queue/346c61e53782b5e0c258f3a62595da0d
>[negroni] Completed 200 OK in 219.385Âµs
>[negroni] Started GET /queue/68f1753cbf11e3a7962681eb39bdfd38
>[negroni] Completed 200 OK in 175.292Âµs
>[kubeexec] DEBUG 2018/06/08 08:43:20 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: sed -i.save "/brick_eab93f37cd264f7b8d5185da3d5e1ff2/d" /var/lib/heketi/fstab
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:43:20 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvs --noheadings --options=thin_count   vg_9394bc70699b006c5460c9f654cf345f/tp_9b2ad28a2bba6499ea37c90a18a2f09e
>
>Result:        0
>[kubeexec] DEBUG 2018/06/08 08:43:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: sed -i.save "/brick_6041796df9e6e8dde7acbc28fb15eebd/d" /var/lib/heketi/fstab
>Result: 
>[negroni] Started GET /queue/346c61e53782b5e0c258f3a62595da0d
>[negroni] Completed 200 OK in 152.352Âµs
>[negroni] Started GET /queue/68f1753cbf11e3a7962681eb39bdfd38
>[negroni] Completed 200 OK in 127.716Âµs
>[kubeexec] DEBUG 2018/06/08 08:43:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvs --noheadings --options=thin_count   vg_d389f0278a774bd7443a09af960961d8/tp_69c43b1a8d9e0514a7e21bde6a2b8a94
>
>Result:        0
>[kubeexec] DEBUG 2018/06/08 08:43:22 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvremove --autobackup=n -f /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_a48060da658afcf0bd672a1d5e325a35
>
>Result:   Logical volume "brick_a48060da658afcf0bd672a1d5e325a35" successfully removed
>[kubeexec] DEBUG 2018/06/08 08:43:22 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvremove --autobackup=n -f /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_92deb78ca12e07cbb3500736c89f75fb
>
>Result:   Logical volume "brick_92deb78ca12e07cbb3500736c89f75fb" successfully removed
>[negroni] Started GET /queue/346c61e53782b5e0c258f3a62595da0d
>[negroni] Completed 200 OK in 103.685Âµs
>[negroni] Started GET /queue/68f1753cbf11e3a7962681eb39bdfd38
>[negroni] Completed 200 OK in 118.673Âµs
>[kubeexec] DEBUG 2018/06/08 08:43:23 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvremove --autobackup=n -f /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_eab93f37cd264f7b8d5185da3d5e1ff2
>
>Result:   Logical volume "brick_eab93f37cd264f7b8d5185da3d5e1ff2" successfully removed
>[kubeexec] DEBUG 2018/06/08 08:43:23 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 07:25:20 UTC; 1h 18min ago
>  Process: 828 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 830 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5cb2299_6aec_11e8_bc4b_005056a5f18a.slice/docker-7f9223abc9a8641cea68f374f4039235515978ee2dab970ac6cbef182c535426.scope/system.slice/glusterd.service
>           ââ  830 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1197 /usr/sbin/glusterfsd -s 10.70.47.76 --volfile-id heketidbstorage.10.70.47.76.var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.76-var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.pid -S /var/run/gluster/b6976f9ddc3abc90e1dfa91c35a7c611.socket --brick-name /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_54c06cdf52c6fe1e3a3f8a9d39cde33d/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.log --xlator-option *-posix.glusterd-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ11949 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/ec468453b11caed5259f5e1abcc65b44.socket --xlator-option *replicate*.node-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94
>
>Jun 08 07:25:17 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 07:25:20 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 08:43:23 Periodic health check status: node d942fe6c0ee5691b7cc263968f97b650 up=true
>[heketi] INFO 2018/06/08 08:43:23 Cleaned 0 nodes from health cache
>[negroni] Started GET /queue/346c61e53782b5e0c258f3a62595da0d
>[negroni] Completed 200 OK in 172.101Âµs
>[kubeexec] DEBUG 2018/06/08 08:43:23 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvremove --autobackup=n -f /dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_6041796df9e6e8dde7acbc28fb15eebd
>
>Result:   Logical volume "brick_6041796df9e6e8dde7acbc28fb15eebd" successfully removed
>[negroni] Started GET /queue/68f1753cbf11e3a7962681eb39bdfd38
>[negroni] Completed 200 OK in 106.554Âµs
>[kubeexec] DEBUG 2018/06/08 08:43:24 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvremove --autobackup=n -f   vg_d389f0278a774bd7443a09af960961d8/tp_69c43b1a8d9e0514a7e21bde6a2b8a94
>
>Result:   Logical volume "tp_69c43b1a8d9e0514a7e21bde6a2b8a94" successfully removed
>[negroni] Started GET /queue/346c61e53782b5e0c258f3a62595da0d
>[negroni] Completed 200 OK in 121.657Âµs
>[kubeexec] DEBUG 2018/06/08 08:43:24 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvremove --autobackup=n -f   vg_9394bc70699b006c5460c9f654cf345f/tp_9b2ad28a2bba6499ea37c90a18a2f09e
>
>Result:   Logical volume "tp_9b2ad28a2bba6499ea37c90a18a2f09e" successfully removed
>[negroni] Started GET /queue/68f1753cbf11e3a7962681eb39bdfd38
>[negroni] Completed 200 OK in 124.879Âµs
>[kubeexec] DEBUG 2018/06/08 08:43:24 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvs --noheadings --options=thin_count   vg_3a4297677881963e3f80124971d50eea/tp_92deb78ca12e07cbb3500736c89f75fb
>
>Result:        0
>[kubeexec] DEBUG 2018/06/08 08:43:25 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvs --noheadings --options=thin_count   vg_d389f0278a774bd7443a09af960961d8/tp_eab93f37cd264f7b8d5185da3d5e1ff2
>
>Result:        0
>[negroni] Started GET /queue/346c61e53782b5e0c258f3a62595da0d
>[negroni] Completed 200 OK in 92.284Âµs
>[kubeexec] DEBUG 2018/06/08 08:43:25 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvs --noheadings --options=thin_count   vg_9394bc70699b006c5460c9f654cf345f/tp_a48060da658afcf0bd672a1d5e325a35
>
>Result:        0
>[negroni] Started GET /queue/68f1753cbf11e3a7962681eb39bdfd38
>[negroni] Completed 200 OK in 89.274Âµs
>[kubeexec] DEBUG 2018/06/08 08:43:25 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvs --noheadings --options=thin_count   vg_66a9af9f6bac95f9d8d556a2f14c29d3/tp_6041796df9e6e8dde7acbc28fb15eebd
>
>Result:        0
>[negroni] Started GET /queue/346c61e53782b5e0c258f3a62595da0d
>[negroni] Completed 200 OK in 151.005Âµs
>[kubeexec] DEBUG 2018/06/08 08:43:26 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: rmdir /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_69c43b1a8d9e0514a7e21bde6a2b8a94
>Result: 
>[negroni] Started GET /queue/68f1753cbf11e3a7962681eb39bdfd38
>[negroni] Completed 200 OK in 96.476Âµs
>[kubeexec] DEBUG 2018/06/08 08:43:26 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: rmdir /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_9b2ad28a2bba6499ea37c90a18a2f09e
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:43:27 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvremove --autobackup=n -f   vg_3a4297677881963e3f80124971d50eea/tp_92deb78ca12e07cbb3500736c89f75fb
>
>Result:   Logical volume "tp_92deb78ca12e07cbb3500736c89f75fb" successfully removed
>[negroni] Started GET /queue/346c61e53782b5e0c258f3a62595da0d
>[negroni] Completed 200 OK in 191.883Âµs
>[negroni] Started GET /queue/68f1753cbf11e3a7962681eb39bdfd38
>[negroni] Completed 200 OK in 180.066Âµs
>[kubeexec] DEBUG 2018/06/08 08:43:27 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvremove --autobackup=n -f   vg_d389f0278a774bd7443a09af960961d8/tp_eab93f37cd264f7b8d5185da3d5e1ff2
>
>Result:   Logical volume "tp_eab93f37cd264f7b8d5185da3d5e1ff2" successfully removed
>[kubeexec] DEBUG 2018/06/08 08:43:28 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvremove --autobackup=n -f   vg_9394bc70699b006c5460c9f654cf345f/tp_a48060da658afcf0bd672a1d5e325a35
>
>Result:   Logical volume "tp_a48060da658afcf0bd672a1d5e325a35" successfully removed
>[kubeexec] DEBUG 2018/06/08 08:43:28 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvremove --autobackup=n -f   vg_66a9af9f6bac95f9d8d556a2f14c29d3/tp_6041796df9e6e8dde7acbc28fb15eebd
>
>Result:   Logical volume "tp_6041796df9e6e8dde7acbc28fb15eebd" successfully removed
>[negroni] Started GET /queue/346c61e53782b5e0c258f3a62595da0d
>[negroni] Completed 200 OK in 123.198Âµs
>[negroni] Started GET /queue/68f1753cbf11e3a7962681eb39bdfd38
>[negroni] Completed 200 OK in 122.281Âµs
>[kubeexec] DEBUG 2018/06/08 08:43:29 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: rmdir /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_eab93f37cd264f7b8d5185da3d5e1ff2
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:43:29 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: rmdir /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_a48060da658afcf0bd672a1d5e325a35
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:43:29 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: rmdir /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_92deb78ca12e07cbb3500736c89f75fb
>Result: 
>[heketi] INFO 2018/06/08 08:43:29 Delete Volume succeeded
>[asynchttp] INFO 2018/06/08 08:43:29 asynchttp.go:292: Completed job 346c61e53782b5e0c258f3a62595da0d in 25.038088296s
>[negroni] Started GET /queue/346c61e53782b5e0c258f3a62595da0d
>[negroni] Completed 204 No Content in 142.118Âµs
>[negroni] Started GET /queue/68f1753cbf11e3a7962681eb39bdfd38
>[negroni] Completed 200 OK in 168.595Âµs
>[kubeexec] DEBUG 2018/06/08 08:43:29 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: rmdir /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_6041796df9e6e8dde7acbc28fb15eebd
>Result: 
>[heketi] INFO 2018/06/08 08:43:29 Delete Volume succeeded
>[asynchttp] INFO 2018/06/08 08:43:29 asynchttp.go:292: Completed job 68f1753cbf11e3a7962681eb39bdfd38 in 25.251210925s
>[negroni] Started GET /queue/68f1753cbf11e3a7962681eb39bdfd38
>[negroni] Completed 204 No Content in 132.801Âµs
>[negroni] Started DELETE /volumes/519804da0a04715f5fe88eef3554e109
>[negroni] Completed 404 Not Found in 4.07577ms
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 234.893Âµs
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 3.082636ms
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 1.193719ms
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 1.159411ms
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 200.224Âµs
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 651.133Âµs
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 570.331Âµs
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 529.642Âµs
>[negroni] Started POST /volumes
>[negroni] Started POST /volumes
>[heketi] INFO 2018/06/08 08:43:31 Allocating brick set #0
>[negroni] Completed 202 Accepted in 14.708355ms
>[asynchttp] INFO 2018/06/08 08:43:31 asynchttp.go:288: Started job 4b8e4d46de1924a143cd62855eaeadee
>[heketi] INFO 2018/06/08 08:43:31 Started async operation: Create Volume
>[negroni] Started GET /queue/4b8e4d46de1924a143cd62855eaeadee
>[negroni] Completed 200 OK in 92.876Âµs
>[heketi] INFO 2018/06/08 08:43:31 Allocating brick set #0
>[heketi] INFO 2018/06/08 08:43:31 Creating brick f05028077b974ebc1f6621aee2184169
>[heketi] INFO 2018/06/08 08:43:31 Creating brick 214eb9006f9103530a1d0310d5f5dcfc
>[heketi] INFO 2018/06/08 08:43:31 Creating brick 4f4d753d298c99eac492c32006c74484
>[negroni] Completed 202 Accepted in 30.237674ms
>[asynchttp] INFO 2018/06/08 08:43:31 asynchttp.go:288: Started job c2abd38f6c55bb468ebcfd89d453fdc9
>[heketi] INFO 2018/06/08 08:43:31 Started async operation: Create Volume
>[negroni] Started GET /queue/c2abd38f6c55bb468ebcfd89d453fdc9
>[negroni] Completed 200 OK in 82.798Âµs
>[heketi] INFO 2018/06/08 08:43:31 Creating brick b38349024b5350e969179b72b5c2af7c
>[heketi] INFO 2018/06/08 08:43:31 Creating brick 9775522b5bb908520213f17390c26d53
>[heketi] INFO 2018/06/08 08:43:31 Creating brick bf9bfa3d8464d1e5476516d891583f10
>[negroni] Started POST /volumes
>[heketi] INFO 2018/06/08 08:43:31 Allocating brick set #0
>[negroni] Completed 202 Accepted in 18.878321ms
>[asynchttp] INFO 2018/06/08 08:43:32 asynchttp.go:288: Started job 8b3ba83547bd02c5fa03488e1de595ea
>[heketi] INFO 2018/06/08 08:43:32 Started async operation: Create Volume
>[negroni] Started GET /queue/8b3ba83547bd02c5fa03488e1de595ea
>[negroni] Completed 200 OK in 105.077Âµs
>[heketi] INFO 2018/06/08 08:43:32 Creating brick 00c2d7c5fda2de77f930386434235209
>[heketi] INFO 2018/06/08 08:43:32 Creating brick 0fb75c3a027f88c53839c0f9578a1801
>[heketi] INFO 2018/06/08 08:43:32 Creating brick 3fce0ef044cb1f17096a5bf85437e0db
>[negroni] Started POST /volumes
>[kubeexec] DEBUG 2018/06/08 08:43:32 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir -p /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_214eb9006f9103530a1d0310d5f5dcfc
>Result: 
>[heketi] INFO 2018/06/08 08:43:32 Allocating brick set #0
>[kubeexec] DEBUG 2018/06/08 08:43:32 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir -p /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_f05028077b974ebc1f6621aee2184169
>Result: 
>[negroni] Completed 202 Accepted in 19.240323ms
>[asynchttp] INFO 2018/06/08 08:43:32 asynchttp.go:288: Started job 5aa78dda299ed08cbb39d86264fe4af8
>[heketi] INFO 2018/06/08 08:43:32 Started async operation: Create Volume
>[negroni] Started GET /queue/5aa78dda299ed08cbb39d86264fe4af8
>[negroni] Completed 200 OK in 99.175Âµs
>[kubeexec] DEBUG 2018/06/08 08:43:32 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir -p /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_4f4d753d298c99eac492c32006c74484
>Result: 
>[heketi] INFO 2018/06/08 08:43:32 Creating brick a40c93ecae170d4b0e28378a7bf7aaee
>[heketi] INFO 2018/06/08 08:43:32 Creating brick 7be0b70e69a87b262dc5f895d408cc1c
>[heketi] INFO 2018/06/08 08:43:32 Creating brick 93847c4b6b28d7ed761f42d934f90771
>[kubeexec] DEBUG 2018/06/08 08:43:32 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_d389f0278a774bd7443a09af960961d8/tp_f05028077b974ebc1f6621aee2184169 --virtualsize 2097152K --name brick_f05028077b974ebc1f6621aee2184169
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_f05028077b974ebc1f6621aee2184169" created.
>[kubeexec] DEBUG 2018/06/08 08:43:32 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_66a9af9f6bac95f9d8d556a2f14c29d3/tp_214eb9006f9103530a1d0310d5f5dcfc --virtualsize 2097152K --name brick_214eb9006f9103530a1d0310d5f5dcfc
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_214eb9006f9103530a1d0310d5f5dcfc" created.
>[kubeexec] DEBUG 2018/06/08 08:43:32 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_96f1667f2f1ced2c5ef94772922be93b/tp_4f4d753d298c99eac492c32006c74484 --virtualsize 2097152K --name brick_4f4d753d298c99eac492c32006c74484
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_4f4d753d298c99eac492c32006c74484" created.
>[kubeexec] DEBUG 2018/06/08 08:43:32 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_f05028077b974ebc1f6621aee2184169
>Result: meta-data=/dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_f05028077b974ebc1f6621aee2184169 isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:43:32 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_4f4d753d298c99eac492c32006c74484
>Result: meta-data=/dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_4f4d753d298c99eac492c32006c74484 isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:43:32 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_214eb9006f9103530a1d0310d5f5dcfc
>Result: meta-data=/dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_214eb9006f9103530a1d0310d5f5dcfc isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:43:32 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: awk "BEGIN {print \"/dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_f05028077b974ebc1f6621aee2184169 /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_f05028077b974ebc1f6621aee2184169 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:43:32 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: awk "BEGIN {print \"/dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_4f4d753d298c99eac492c32006c74484 /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_4f4d753d298c99eac492c32006c74484 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:43:32 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: awk "BEGIN {print \"/dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_214eb9006f9103530a1d0310d5f5dcfc /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_214eb9006f9103530a1d0310d5f5dcfc xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:43:32 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_f05028077b974ebc1f6621aee2184169 /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_f05028077b974ebc1f6621aee2184169
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:43:32 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_214eb9006f9103530a1d0310d5f5dcfc /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_214eb9006f9103530a1d0310d5f5dcfc
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:43:32 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_4f4d753d298c99eac492c32006c74484 /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_4f4d753d298c99eac492c32006c74484
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:43:32 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_f05028077b974ebc1f6621aee2184169/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:43:32 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_214eb9006f9103530a1d0310d5f5dcfc/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:43:32 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_4f4d753d298c99eac492c32006c74484/brick
>Result: 
>[negroni] Started GET /queue/4b8e4d46de1924a143cd62855eaeadee
>[negroni] Completed 200 OK in 135.103Âµs
>[negroni] Started GET /queue/c2abd38f6c55bb468ebcfd89d453fdc9
>[negroni] Completed 200 OK in 166.332Âµs
>[kubeexec] DEBUG 2018/06/08 08:43:32 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: chown :2000 /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_214eb9006f9103530a1d0310d5f5dcfc/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:43:32 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: chown :2000 /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_f05028077b974ebc1f6621aee2184169/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:43:32 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: chown :2000 /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_4f4d753d298c99eac492c32006c74484/brick
>Result: 
>[negroni] Started GET /queue/8b3ba83547bd02c5fa03488e1de595ea
>[negroni] Completed 200 OK in 96.794Âµs
>[negroni] Started GET /queue/5aa78dda299ed08cbb39d86264fe4af8
>[negroni] Completed 200 OK in 116.049Âµs
>[kubeexec] DEBUG 2018/06/08 08:43:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: chmod 2775 /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_f05028077b974ebc1f6621aee2184169/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:43:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: chmod 2775 /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_214eb9006f9103530a1d0310d5f5dcfc/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:43:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: chmod 2775 /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_4f4d753d298c99eac492c32006c74484/brick
>Result: 
>[cmdexec] INFO 2018/06/08 08:43:33 Creating volume vol_5db55c483f6984d9917cfe4b3c8b3cbc replica 3
>[kubeexec] DEBUG 2018/06/08 08:43:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir -p /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_9775522b5bb908520213f17390c26d53
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:43:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir -p /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_bf9bfa3d8464d1e5476516d891583f10
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:43:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir -p /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_b38349024b5350e969179b72b5c2af7c
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:43:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_b861af9cc67f5bf8d5a8f9e38f56b14e/tp_9775522b5bb908520213f17390c26d53 --virtualsize 2097152K --name brick_9775522b5bb908520213f17390c26d53
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_9775522b5bb908520213f17390c26d53" created.
>[kubeexec] DEBUG 2018/06/08 08:43:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_3a4297677881963e3f80124971d50eea/tp_bf9bfa3d8464d1e5476516d891583f10 --virtualsize 2097152K --name brick_bf9bfa3d8464d1e5476516d891583f10
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_bf9bfa3d8464d1e5476516d891583f10" created.
>[kubeexec] DEBUG 2018/06/08 08:43:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_96f1667f2f1ced2c5ef94772922be93b/tp_b38349024b5350e969179b72b5c2af7c --virtualsize 2097152K --name brick_b38349024b5350e969179b72b5c2af7c
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_b38349024b5350e969179b72b5c2af7c" created.
>[kubeexec] DEBUG 2018/06/08 08:43:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_9775522b5bb908520213f17390c26d53
>Result: meta-data=/dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_9775522b5bb908520213f17390c26d53 isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:43:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_b38349024b5350e969179b72b5c2af7c
>Result: meta-data=/dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_b38349024b5350e969179b72b5c2af7c isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:43:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: awk "BEGIN {print \"/dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_9775522b5bb908520213f17390c26d53 /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_9775522b5bb908520213f17390c26d53 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:43:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_bf9bfa3d8464d1e5476516d891583f10
>Result: meta-data=/dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_bf9bfa3d8464d1e5476516d891583f10 isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:43:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: awk "BEGIN {print \"/dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_b38349024b5350e969179b72b5c2af7c /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_b38349024b5350e969179b72b5c2af7c xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:43:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_9775522b5bb908520213f17390c26d53 /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_9775522b5bb908520213f17390c26d53
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:43:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: awk "BEGIN {print \"/dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_bf9bfa3d8464d1e5476516d891583f10 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_bf9bfa3d8464d1e5476516d891583f10 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:43:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_b38349024b5350e969179b72b5c2af7c /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_b38349024b5350e969179b72b5c2af7c
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:43:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_9775522b5bb908520213f17390c26d53/brick
>Result: 
>[negroni] Started GET /queue/4b8e4d46de1924a143cd62855eaeadee
>[negroni] Completed 200 OK in 112.388Âµs
>[negroni] Started GET /queue/c2abd38f6c55bb468ebcfd89d453fdc9
>[negroni] Completed 200 OK in 171.584Âµs
>[kubeexec] DEBUG 2018/06/08 08:43:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_bf9bfa3d8464d1e5476516d891583f10 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_bf9bfa3d8464d1e5476516d891583f10
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:43:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_b38349024b5350e969179b72b5c2af7c/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:43:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: chown :2001 /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_9775522b5bb908520213f17390c26d53/brick
>Result: 
>[negroni] Started GET /queue/8b3ba83547bd02c5fa03488e1de595ea
>[negroni] Completed 200 OK in 110.21Âµs
>[kubeexec] DEBUG 2018/06/08 08:43:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_bf9bfa3d8464d1e5476516d891583f10/brick
>Result: 
>[negroni] Started GET /queue/5aa78dda299ed08cbb39d86264fe4af8
>[negroni] Completed 200 OK in 96.528Âµs
>[kubeexec] DEBUG 2018/06/08 08:43:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: chmod 2775 /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_9775522b5bb908520213f17390c26d53/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:43:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: chown :2001 /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_b38349024b5350e969179b72b5c2af7c/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:43:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: chown :2001 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_bf9bfa3d8464d1e5476516d891583f10/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:43:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: chmod 2775 /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_b38349024b5350e969179b72b5c2af7c/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:43:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir -p /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_3fce0ef044cb1f17096a5bf85437e0db
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:43:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: chmod 2775 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_bf9bfa3d8464d1e5476516d891583f10/brick
>Result: 
>[cmdexec] INFO 2018/06/08 08:43:34 Creating volume vol_a4a6e4892da299f6c5634b8a2def697e replica 3
>[kubeexec] DEBUG 2018/06/08 08:43:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir -p /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_0fb75c3a027f88c53839c0f9578a1801
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:43:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir -p /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_00c2d7c5fda2de77f930386434235209
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:43:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_d389f0278a774bd7443a09af960961d8/tp_3fce0ef044cb1f17096a5bf85437e0db --virtualsize 2097152K --name brick_3fce0ef044cb1f17096a5bf85437e0db
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_3fce0ef044cb1f17096a5bf85437e0db" created.
>[kubeexec] DEBUG 2018/06/08 08:43:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_9394bc70699b006c5460c9f654cf345f/tp_0fb75c3a027f88c53839c0f9578a1801 --virtualsize 2097152K --name brick_0fb75c3a027f88c53839c0f9578a1801
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_0fb75c3a027f88c53839c0f9578a1801" created.
>[kubeexec] DEBUG 2018/06/08 08:43:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_3fce0ef044cb1f17096a5bf85437e0db
>Result: meta-data=/dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_3fce0ef044cb1f17096a5bf85437e0db isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:43:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_3a4297677881963e3f80124971d50eea/tp_00c2d7c5fda2de77f930386434235209 --virtualsize 2097152K --name brick_00c2d7c5fda2de77f930386434235209
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_00c2d7c5fda2de77f930386434235209" created.
>[kubeexec] DEBUG 2018/06/08 08:43:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: awk "BEGIN {print \"/dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_3fce0ef044cb1f17096a5bf85437e0db /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_3fce0ef044cb1f17096a5bf85437e0db xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:43:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_0fb75c3a027f88c53839c0f9578a1801
>Result: meta-data=/dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_0fb75c3a027f88c53839c0f9578a1801 isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:43:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_00c2d7c5fda2de77f930386434235209
>Result: meta-data=/dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_00c2d7c5fda2de77f930386434235209 isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:43:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: awk "BEGIN {print \"/dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_0fb75c3a027f88c53839c0f9578a1801 /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_0fb75c3a027f88c53839c0f9578a1801 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:43:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: awk "BEGIN {print \"/dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_00c2d7c5fda2de77f930386434235209 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_00c2d7c5fda2de77f930386434235209 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:43:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_3fce0ef044cb1f17096a5bf85437e0db /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_3fce0ef044cb1f17096a5bf85437e0db
>Result: 
>[negroni] Started GET /queue/4b8e4d46de1924a143cd62855eaeadee
>[negroni] Completed 200 OK in 121.257Âµs
>[negroni] Started GET /queue/c2abd38f6c55bb468ebcfd89d453fdc9
>[negroni] Completed 200 OK in 138.188Âµs
>[kubeexec] DEBUG 2018/06/08 08:43:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_3fce0ef044cb1f17096a5bf85437e0db/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:43:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_0fb75c3a027f88c53839c0f9578a1801 /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_0fb75c3a027f88c53839c0f9578a1801
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:43:35 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_00c2d7c5fda2de77f930386434235209 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_00c2d7c5fda2de77f930386434235209
>Result: 
>[negroni] Started GET /queue/8b3ba83547bd02c5fa03488e1de595ea
>[negroni] Completed 200 OK in 144.592Âµs
>[negroni] Started GET /queue/5aa78dda299ed08cbb39d86264fe4af8
>[negroni] Completed 200 OK in 110.753Âµs
>[kubeexec] DEBUG 2018/06/08 08:43:35 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: chown :2002 /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_3fce0ef044cb1f17096a5bf85437e0db/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:43:35 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_00c2d7c5fda2de77f930386434235209/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:43:35 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_0fb75c3a027f88c53839c0f9578a1801/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:43:35 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: chmod 2775 /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_3fce0ef044cb1f17096a5bf85437e0db/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:43:35 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: chown :2002 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_00c2d7c5fda2de77f930386434235209/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:43:35 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: chown :2002 /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_0fb75c3a027f88c53839c0f9578a1801/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:43:35 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir -p /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_a40c93ecae170d4b0e28378a7bf7aaee
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:43:35 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: chmod 2775 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_00c2d7c5fda2de77f930386434235209/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:43:35 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: chmod 2775 /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_0fb75c3a027f88c53839c0f9578a1801/brick
>Result: 
>[cmdexec] INFO 2018/06/08 08:43:35 Creating volume vol_966a92ed3e4374a5e634f7b133c49e52 replica 3
>[kubeexec] DEBUG 2018/06/08 08:43:35 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir -p /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_93847c4b6b28d7ed761f42d934f90771
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:43:35 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir -p /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_7be0b70e69a87b262dc5f895d408cc1c
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:43:35 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_b861af9cc67f5bf8d5a8f9e38f56b14e/tp_a40c93ecae170d4b0e28378a7bf7aaee --virtualsize 2097152K --name brick_a40c93ecae170d4b0e28378a7bf7aaee
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_a40c93ecae170d4b0e28378a7bf7aaee" created.
>[kubeexec] DEBUG 2018/06/08 08:43:35 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_a40c93ecae170d4b0e28378a7bf7aaee
>Result: meta-data=/dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_a40c93ecae170d4b0e28378a7bf7aaee isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:43:35 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_3a4297677881963e3f80124971d50eea/tp_93847c4b6b28d7ed761f42d934f90771 --virtualsize 2097152K --name brick_93847c4b6b28d7ed761f42d934f90771
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_93847c4b6b28d7ed761f42d934f90771" created.
>[kubeexec] DEBUG 2018/06/08 08:43:35 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_96f1667f2f1ced2c5ef94772922be93b/tp_7be0b70e69a87b262dc5f895d408cc1c --virtualsize 2097152K --name brick_7be0b70e69a87b262dc5f895d408cc1c
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_7be0b70e69a87b262dc5f895d408cc1c" created.
>[kubeexec] DEBUG 2018/06/08 08:43:35 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: awk "BEGIN {print \"/dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_a40c93ecae170d4b0e28378a7bf7aaee /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_a40c93ecae170d4b0e28378a7bf7aaee xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:43:35 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_93847c4b6b28d7ed761f42d934f90771
>Result: meta-data=/dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_93847c4b6b28d7ed761f42d934f90771 isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[negroni] Started GET /queue/4b8e4d46de1924a143cd62855eaeadee
>[negroni] Completed 200 OK in 122.854Âµs
>[kubeexec] DEBUG 2018/06/08 08:43:35 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: awk "BEGIN {print \"/dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_93847c4b6b28d7ed761f42d934f90771 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_93847c4b6b28d7ed761f42d934f90771 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[negroni] Started GET /queue/c2abd38f6c55bb468ebcfd89d453fdc9
>[negroni] Completed 200 OK in 123.429Âµs
>[kubeexec] DEBUG 2018/06/08 08:43:35 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_a40c93ecae170d4b0e28378a7bf7aaee /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_a40c93ecae170d4b0e28378a7bf7aaee
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:43:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_7be0b70e69a87b262dc5f895d408cc1c
>Result: meta-data=/dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_7be0b70e69a87b262dc5f895d408cc1c isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[negroni] Started GET /queue/8b3ba83547bd02c5fa03488e1de595ea
>[negroni] Completed 200 OK in 101.176Âµs
>[negroni] Started GET /queue/5aa78dda299ed08cbb39d86264fe4af8
>[negroni] Completed 200 OK in 147.794Âµs
>[kubeexec] DEBUG 2018/06/08 08:43:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_a40c93ecae170d4b0e28378a7bf7aaee/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:43:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: awk "BEGIN {print \"/dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_7be0b70e69a87b262dc5f895d408cc1c /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_7be0b70e69a87b262dc5f895d408cc1c xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:43:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_93847c4b6b28d7ed761f42d934f90771 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_93847c4b6b28d7ed761f42d934f90771
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:43:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: chown :2003 /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_a40c93ecae170d4b0e28378a7bf7aaee/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:43:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_93847c4b6b28d7ed761f42d934f90771/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:43:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_7be0b70e69a87b262dc5f895d408cc1c /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_7be0b70e69a87b262dc5f895d408cc1c
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:43:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: chmod 2775 /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_a40c93ecae170d4b0e28378a7bf7aaee/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:43:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: chown :2003 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_93847c4b6b28d7ed761f42d934f90771/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:43:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_7be0b70e69a87b262dc5f895d408cc1c/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:43:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: chmod 2775 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_93847c4b6b28d7ed761f42d934f90771/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:43:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: chown :2003 /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_7be0b70e69a87b262dc5f895d408cc1c/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:43:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: chmod 2775 /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_7be0b70e69a87b262dc5f895d408cc1c/brick
>Result: 
>[cmdexec] INFO 2018/06/08 08:43:36 Creating volume vol_9c85de66c12db0a72b4d16fe888ff74d replica 3
>[kubeexec] DEBUG 2018/06/08 08:43:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: gluster --mode=script volume create vol_5db55c483f6984d9917cfe4b3c8b3cbc replica 3 10.70.46.187:/var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_f05028077b974ebc1f6621aee2184169/brick 10.70.46.122:/var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_214eb9006f9103530a1d0310d5f5dcfc/brick 10.70.47.76:/var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_4f4d753d298c99eac492c32006c74484/brick
>Result: volume create: vol_5db55c483f6984d9917cfe4b3c8b3cbc: success: please start the volume to access data
>[kubeexec] DEBUG 2018/06/08 08:43:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script volume create vol_966a92ed3e4374a5e634f7b133c49e52 replica 3 10.70.46.122:/var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_00c2d7c5fda2de77f930386434235209/brick 10.70.47.76:/var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_0fb75c3a027f88c53839c0f9578a1801/brick 10.70.46.187:/var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_3fce0ef044cb1f17096a5bf85437e0db/brick
>Result: volume create: vol_966a92ed3e4374a5e634f7b133c49e52: success: please start the volume to access data
>[kubeexec] DEBUG 2018/06/08 08:43:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script volume create vol_a4a6e4892da299f6c5634b8a2def697e replica 3 10.70.47.76:/var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_b38349024b5350e969179b72b5c2af7c/brick 10.70.46.187:/var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_9775522b5bb908520213f17390c26d53/brick 10.70.46.122:/var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_bf9bfa3d8464d1e5476516d891583f10/brick
>Result: volume create: vol_a4a6e4892da299f6c5634b8a2def697e: success: please start the volume to access data
>[negroni] Started GET /queue/4b8e4d46de1924a143cd62855eaeadee
>[negroni] Started GET /queue/c2abd38f6c55bb468ebcfd89d453fdc9
>[negroni] Completed 200 OK in 105.218Âµs
>[negroni] Completed 200 OK in 72.027Âµs
>[negroni] Started GET /queue/8b3ba83547bd02c5fa03488e1de595ea
>[negroni] Completed 200 OK in 164.954Âµs
>[negroni] Started GET /queue/5aa78dda299ed08cbb39d86264fe4af8
>[negroni] Completed 200 OK in 113.203Âµs
>[negroni] Started GET /queue/4b8e4d46de1924a143cd62855eaeadee
>[negroni] Completed 200 OK in 190.172Âµs
>[negroni] Started GET /queue/c2abd38f6c55bb468ebcfd89d453fdc9
>[negroni] Completed 200 OK in 358.473Âµs
>[negroni] Started GET /queue/8b3ba83547bd02c5fa03488e1de595ea
>[negroni] Completed 200 OK in 103.274Âµs
>[negroni] Started GET /queue/5aa78dda299ed08cbb39d86264fe4af8
>[negroni] Completed 200 OK in 109.645Âµs
>[kubeexec] DEBUG 2018/06/08 08:43:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: gluster --mode=script volume start vol_5db55c483f6984d9917cfe4b3c8b3cbc
>Result: volume start: vol_5db55c483f6984d9917cfe4b3c8b3cbc: success
>[heketi] INFO 2018/06/08 08:43:38 Create Volume succeeded
>[asynchttp] INFO 2018/06/08 08:43:38 asynchttp.go:292: Completed job 4b8e4d46de1924a143cd62855eaeadee in 6.974262608s
>[negroni] Started GET /queue/4b8e4d46de1924a143cd62855eaeadee
>[negroni] Completed 303 See Other in 117.468Âµs
>[negroni] Started GET /queue/c2abd38f6c55bb468ebcfd89d453fdc9
>[negroni] Completed 200 OK in 54.349Âµs
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 4.956864ms
>[negroni] Started GET /clusters/0a73c60efdd4673113b668afea101e6d
>[negroni] Completed 200 OK in 1.706283ms
>[negroni] Started GET /nodes/278bd6b4e16a8e62ef15aaae22e6abc1
>[negroni] Completed 200 OK in 2.706513ms
>[negroni] Started GET /nodes/70423cc0bcd044fe5ba8bbbd256a3e49
>[negroni] Completed 200 OK in 1.846939ms
>[negroni] Started GET /nodes/d942fe6c0ee5691b7cc263968f97b650
>[negroni] Completed 200 OK in 926.005Âµs
>[negroni] Started GET /queue/8b3ba83547bd02c5fa03488e1de595ea
>[negroni] Completed 200 OK in 155.084Âµs
>[negroni] Started GET /queue/5aa78dda299ed08cbb39d86264fe4af8
>[negroni] Completed 200 OK in 169.555Âµs
>[negroni] Started GET /queue/c2abd38f6c55bb468ebcfd89d453fdc9
>[negroni] Completed 200 OK in 105.106Âµs
>[negroni] Started GET /queue/8b3ba83547bd02c5fa03488e1de595ea
>[negroni] Completed 200 OK in 126.275Âµs
>[negroni] Started GET /queue/5aa78dda299ed08cbb39d86264fe4af8
>[negroni] Completed 200 OK in 124.025Âµs
>[negroni] Started GET /queue/c2abd38f6c55bb468ebcfd89d453fdc9
>[negroni] Completed 200 OK in 107.003Âµs
>[negroni] Started GET /queue/8b3ba83547bd02c5fa03488e1de595ea
>[negroni] Completed 200 OK in 106.63Âµs
>[negroni] Started GET /queue/5aa78dda299ed08cbb39d86264fe4af8
>[negroni] Completed 200 OK in 109.519Âµs
>[negroni] Started GET /queue/c2abd38f6c55bb468ebcfd89d453fdc9
>[negroni] Completed 200 OK in 119.192Âµs
>[negroni] Started GET /queue/8b3ba83547bd02c5fa03488e1de595ea
>[negroni] Completed 200 OK in 120.306Âµs
>[negroni] Started GET /queue/5aa78dda299ed08cbb39d86264fe4af8
>[negroni] Completed 200 OK in 81.991Âµs
>[negroni] Started GET /queue/c2abd38f6c55bb468ebcfd89d453fdc9
>[negroni] Completed 200 OK in 145.414Âµs
>[negroni] Started GET /queue/8b3ba83547bd02c5fa03488e1de595ea
>[negroni] Completed 200 OK in 135.486Âµs
>[negroni] Started GET /queue/5aa78dda299ed08cbb39d86264fe4af8
>[negroni] Completed 200 OK in 124.179Âµs
>[kubeexec] DEBUG 2018/06/08 08:43:43 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script volume start vol_a4a6e4892da299f6c5634b8a2def697e
>Result: volume start: vol_a4a6e4892da299f6c5634b8a2def697e: success
>[kubeexec] DEBUG 2018/06/08 08:43:43 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: gluster --mode=script volume create vol_9c85de66c12db0a72b4d16fe888ff74d replica 3 10.70.46.187:/var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_a40c93ecae170d4b0e28378a7bf7aaee/brick 10.70.46.122:/var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_93847c4b6b28d7ed761f42d934f90771/brick 10.70.47.76:/var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_7be0b70e69a87b262dc5f895d408cc1c/brick
>Result: volume create: vol_9c85de66c12db0a72b4d16fe888ff74d: success: please start the volume to access data
>[kubeexec] DEBUG 2018/06/08 08:43:43 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script volume start vol_966a92ed3e4374a5e634f7b133c49e52
>Result: volume start: vol_966a92ed3e4374a5e634f7b133c49e52: success
>[heketi] INFO 2018/06/08 08:43:43 Create Volume succeeded
>[asynchttp] INFO 2018/06/08 08:43:43 asynchttp.go:292: Completed job c2abd38f6c55bb468ebcfd89d453fdc9 in 12.043553614s
>[heketi] INFO 2018/06/08 08:43:43 Create Volume succeeded
>[asynchttp] INFO 2018/06/08 08:43:43 asynchttp.go:292: Completed job 8b3ba83547bd02c5fa03488e1de595ea in 11.970530637s
>[negroni] Started GET /queue/c2abd38f6c55bb468ebcfd89d453fdc9
>[negroni] Completed 303 See Other in 175.424Âµs
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 3.966686ms
>[negroni] Started GET /clusters/0a73c60efdd4673113b668afea101e6d
>[negroni] Completed 200 OK in 326.02Âµs
>[negroni] Started GET /nodes/278bd6b4e16a8e62ef15aaae22e6abc1
>[negroni] Completed 200 OK in 3.076164ms
>[negroni] Started GET /nodes/70423cc0bcd044fe5ba8bbbd256a3e49
>[negroni] Completed 200 OK in 1.189707ms
>[negroni] Started GET /nodes/d942fe6c0ee5691b7cc263968f97b650
>[negroni] Completed 200 OK in 852.583Âµs
>[negroni] Started GET /queue/8b3ba83547bd02c5fa03488e1de595ea
>[negroni] Completed 303 See Other in 150.576Âµs
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 2.000867ms
>[negroni] Started GET /clusters/0a73c60efdd4673113b668afea101e6d
>[negroni] Completed 200 OK in 651.682Âµs
>[negroni] Started GET /nodes/278bd6b4e16a8e62ef15aaae22e6abc1
>[negroni] Completed 200 OK in 1.215816ms
>[negroni] Started GET /nodes/70423cc0bcd044fe5ba8bbbd256a3e49
>[negroni] Completed 200 OK in 1.373329ms
>[negroni] Started GET /nodes/d942fe6c0ee5691b7cc263968f97b650
>[negroni] Completed 200 OK in 1.043432ms
>[negroni] Started GET /queue/5aa78dda299ed08cbb39d86264fe4af8
>[negroni] Completed 200 OK in 113.014Âµs
>[negroni] Started GET /queue/5aa78dda299ed08cbb39d86264fe4af8
>[negroni] Completed 200 OK in 189.883Âµs
>[negroni] Started GET /queue/5aa78dda299ed08cbb39d86264fe4af8
>[negroni] Completed 200 OK in 195.022Âµs
>[negroni] Started GET /queue/5aa78dda299ed08cbb39d86264fe4af8
>[negroni] Completed 200 OK in 113.379Âµs
>[kubeexec] DEBUG 2018/06/08 08:43:47 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: gluster --mode=script volume start vol_9c85de66c12db0a72b4d16fe888ff74d
>Result: volume start: vol_9c85de66c12db0a72b4d16fe888ff74d: success
>[heketi] INFO 2018/06/08 08:43:47 Create Volume succeeded
>[asynchttp] INFO 2018/06/08 08:43:47 asynchttp.go:292: Completed job 5aa78dda299ed08cbb39d86264fe4af8 in 15.197281898s
>[negroni] Started GET /queue/5aa78dda299ed08cbb39d86264fe4af8
>[negroni] Completed 303 See Other in 277.638Âµs
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 5.052531ms
>[negroni] Started GET /clusters/0a73c60efdd4673113b668afea101e6d
>[negroni] Completed 200 OK in 330.844Âµs
>[negroni] Started GET /nodes/278bd6b4e16a8e62ef15aaae22e6abc1
>[negroni] Completed 200 OK in 3.654089ms
>[negroni] Started GET /nodes/70423cc0bcd044fe5ba8bbbd256a3e49
>[negroni] Completed 200 OK in 1.439958ms
>[negroni] Started GET /nodes/d942fe6c0ee5691b7cc263968f97b650
>[negroni] Completed 200 OK in 1.535745ms
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 2.538844ms
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 1.062205ms
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 1.312156ms
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 2.361872ms
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 958.309Âµs
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 608.712Âµs
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 617.882Âµs
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 646.398Âµs
>[negroni] Started DELETE /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 202 Accepted in 9.22408ms
>[asynchttp] INFO 2018/06/08 08:43:49 asynchttp.go:288: Started job a9e6acc2efeee7febda192bc9c56b54e
>[heketi] INFO 2018/06/08 08:43:49 Started async operation: Delete Volume
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 127.611Âµs
>[kubeexec] DEBUG 2018/06/08 08:43:49 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script snapshot list vol_9c85de66c12db0a72b4d16fe888ff74d --xml
>Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
><cliOutput>
>  <opRet>0</opRet>
>  <opErrno>0</opErrno>
>  <opErrstr/>
>  <snapList>
>    <count>0</count>
>  </snapList>
></cliOutput>
>[negroni] Started DELETE /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 202 Accepted in 12.841174ms
>[asynchttp] INFO 2018/06/08 08:43:49 asynchttp.go:288: Started job f57802d82a15df4af9a77cc5ba0a7516
>[heketi] INFO 2018/06/08 08:43:49 Started async operation: Delete Volume
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 124.402Âµs
>[kubeexec] DEBUG 2018/06/08 08:43:50 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script snapshot list vol_966a92ed3e4374a5e634f7b133c49e52 --xml
>Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
><cliOutput>
>  <opRet>0</opRet>
>  <opErrno>0</opErrno>
>  <opErrstr/>
>  <snapList>
>    <count>0</count>
>  </snapList>
></cliOutput>
>[negroni] Started DELETE /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 202 Accepted in 14.556322ms
>[asynchttp] INFO 2018/06/08 08:43:50 asynchttp.go:288: Started job 9e1718475edb952ba52dafee886587f8
>[heketi] INFO 2018/06/08 08:43:50 Started async operation: Delete Volume
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 98.953Âµs
>[kubeexec] DEBUG 2018/06/08 08:43:50 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: gluster --mode=script snapshot list vol_a4a6e4892da299f6c5634b8a2def697e --xml
>Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
><cliOutput>
>  <opRet>0</opRet>
>  <opErrno>0</opErrno>
>  <opErrstr/>
>  <snapList>
>    <count>0</count>
>  </snapList>
></cliOutput>
>[negroni] Started DELETE /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 202 Accepted in 14.948979ms
>[asynchttp] INFO 2018/06/08 08:43:50 asynchttp.go:288: Started job c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[heketi] INFO 2018/06/08 08:43:50 Started async operation: Delete Volume
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 124.223Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 207.412Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 179.674Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 112.11Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 229.202Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 102.411Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 133.541Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 177.485Âµs
>[kubeexec] DEBUG 2018/06/08 08:43:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script volume stop vol_9c85de66c12db0a72b4d16fe888ff74d force
>Result: volume stop: vol_9c85de66c12db0a72b4d16fe888ff74d: success
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 110.846Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 226.88Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 237.808Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 219.919Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 156.179Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 121.306Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 109.043Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 174.997Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 140.505Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 156.011Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 148.046Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 120.631Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 133.628Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 209.746Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 120.355Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 207.735Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 212.109Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 252.651Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 256.095Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 222.014Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 138.624Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 169.923Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 146.494Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 161.973Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 200.904Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 189.675Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 153.76Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 300.024Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 320.017Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 328.938Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 160.973Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 215.252Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 167.169Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 241.928Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 165.559Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 171.522Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 232.389Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 166.223Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 241.272Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 151.22Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 169.052Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 130.712Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 257.041Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 232.255Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 170.155Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 226.248Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 216.674Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 217.629Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 124.54Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 217.397Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 159.813Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 362.377Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 168.306Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 319.332Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 160.194Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 133.269Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 161.241Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 181.182Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 227.309Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 182.954Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 341.967Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 150.522Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 133.413Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 191.721Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 174.652Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 251.544Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 135.366Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 151.117Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 316.698Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 169.152Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 260.728Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 189.76Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 184.871Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 290.991Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 126.134Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 224.298Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 215.463Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 156.787Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 266.745Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 160.754Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 182.809Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 179.125Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 149.954Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 128.896Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 144.467Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 201.6Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 158.302Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 122.665Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 259.282Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 243.035Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 229.518Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 213.478Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 242.858Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 329.618Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 178.859Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 131.523Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 127.52Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 147.927Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 141.333Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 137.095Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 232.381Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 228.415Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 161.679Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 210.048Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 123.715Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 135.003Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 210.141Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 175.529Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 200.957Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 222.491Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 184.896Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 159.19Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 271.415Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 305.374Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 193.436Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 297.629Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 181.338Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 333.244Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 162.359Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 236.324Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 143.506Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 224.423Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 202.974Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 300.438Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 214.945Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 229.168Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 252.732Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 206.972Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 184.643Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 293.681Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 158.177Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 169.658Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 209.835Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 305.28Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 144.001Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 174.08Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 121.724Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 279.108Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 452.384Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 205.316Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 182.169Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 278.914Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 230.077Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 217.995Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 182.534Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 340.678Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 218.938Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 154.049Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 159.909Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 261.655Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 189.189Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 168.218Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 196.873Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 181.839Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 253.325Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 126.594Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 192.685Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 147.446Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 152.24Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 131.437Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 250.551Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 143.795Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 207.975Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 238.858Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 385.164Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 193.76Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 204.528Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 177.637Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 196.732Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 204.713Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 167.677Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 201.623Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 206.295Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 243.645Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 269.058Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 190.169Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 174.773Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 312.301Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 192.505Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 214.602Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 138.957Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 239.626Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 143.753Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 210.859Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 155.345Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 146.727Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 132.845Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 269.045Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 148.1Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 219.148Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 189.071Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 118.583Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 224.935Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 288.851Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 288.073Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 266.641Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 260.13Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 138.836Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 168.518Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 191.066Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 244.764Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 141.441Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 127.683Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 223.742Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 171.606Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 186.196Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 132.045Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 117.983Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 171.095Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 125.475Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 250.191Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 117.397Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 270.057Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 169.759Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 235.021Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 334.088Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 236.938Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 125.338Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 157.496Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 210.669Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 262.948Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 340.429Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 234.945Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 294.998Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 197.205Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 167.126Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 175.974Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 200.472Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 159.507Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 244.934Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 183.159Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 155.379Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 197.546Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 185.506Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 175.429Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 205.362Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 277.448Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 182.524Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 143.032Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 259.746Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 220.042Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 230.652Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 130.256Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 156.32Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 169.678Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 279.391Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 209.581Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 163.748Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 160.806Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 316.521Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 225.071Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 174.254Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 181.541Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 208.052Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 176.414Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 203.325Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 189.769Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 195.869Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 210.024Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 236.465Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 199.702Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 246.605Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 224.245Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 163.929Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 131.199Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 121.321Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 205.462Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 176.089Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 199.786Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 186.739Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 241.805Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 224.875Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 186.049Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 217.345Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 195.425Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 196.676Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 184.162Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 155.239Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 135.218Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 195.277Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 269.497Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 179.608Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 195.632Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 197.485Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 182.045Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 168.938Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 160.899Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 204.996Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 172.337Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 177.422Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 164.898Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 182.324Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 130.044Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 164.738Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 272.134Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 198.461Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 169.24Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 300.231Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 126.726Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 331.957Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 156.487Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 143.978Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 173.842Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 288.765Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 187.974Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 211.11Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 171.129Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 193.861Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 184.399Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 204.611Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 138.189Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 170.665Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 213.938Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 136.423Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 195.62Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 187.969Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 230.758Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 125.096Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 137.442Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 184.252Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 152.421Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 139.175Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 154.385Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 192.699Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 260.685Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 135.279Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 134.615Âµs
>[heketi] INFO 2018/06/08 08:45:14 Starting Node Health Status refresh
>[cmdexec] INFO 2018/06/08 08:45:14 Check Glusterd service status in node dhcp46-187.lab.eng.blr.redhat.com
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 218.605Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 178.972Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 257.328Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 201.894Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 189.896Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 196.585Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 166.128Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 186.989Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 151.383Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 202.835Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 206.771Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 154.105Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 144.823Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 152.133Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 192.308Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 140.64Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 180.522Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 181.926Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 138.539Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 168.981Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 223.706Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 138.241Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 189.362Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 178.769Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 136.853Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 229.081Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 143.317Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 132.588Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 180.339Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 127.46Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 195.921Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 128.754Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 142.627Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 178.137Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 140.397Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 204.489Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 198.329Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 198.333Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 187.765Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 195.249Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 130.032Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 156.025Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 254.158Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 133.087Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 144.057Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 193.479Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 186.359Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 138.279Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 214.829Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 251.429Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 208.315Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 198.468Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 171.336Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 228.726Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 223.328Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 179.538Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 128.972Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 134.495Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 118.465Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 178.701Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 202.722Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 195.119Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 256.945Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 191.476Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 188.015Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 236.265Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 236.444Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 160.603Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 149.356Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 191.085Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 165.162Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 144.014Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 333.133Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 200.586Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 188.358Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 199.798Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 137.359Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 187.492Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 199.585Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 133.154Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 162.62Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 189.394Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 161.227Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 206.792Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 193.912Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 212.03Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 207.092Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 210.206Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 143.664Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 182.162Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 206.222Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 176.659Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 174.972Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 180.171Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 264.185Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 187.402Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 177.153Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 203.531Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 138.63Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 116.726Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 145.239Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 127.347Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 173.372Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 190.532Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 166.086Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 179.068Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 127.077Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 183.163Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 128.977Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 186.929Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 123.28Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 277.78Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 185.222Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 183.236Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 214.844Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 195.955Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 207.345Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 197.841Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 190.552Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 221.899Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 153.377Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 136.719Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 134.813Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 284.555Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 231.605Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 221.096Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 178.222Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 183.983Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 128.069Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 169.501Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 199.142Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 178.525Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 149.048Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 197.282Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 177.162Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 207.103Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 171.659Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 268.171Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 200.662Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 168.539Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 162.731Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 177.207Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 160.013Âµs
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 200 OK in 194.973Âµs
>[kubeexec] ERROR 2018/06/08 08:45:50 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume stop vol_966a92ed3e4374a5e634f7b133c49e52 force] on glusterfs-storage-pg4xc: Err[command terminated with exit code 1]: Stdout [Error : Request timed out
>]: Stderr []
>[cmdexec] ERROR 2018/06/08 08:45:50 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:144: Unable to stop volume vol_966a92ed3e4374a5e634f7b133c49e52: Unable to execute command on glusterfs-storage-pg4xc: 
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 200 OK in 262.561Âµs
>[kubeexec] DEBUG 2018/06/08 08:45:50 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script snapshot list vol_5db55c483f6984d9917cfe4b3c8b3cbc --xml
>Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
><cliOutput>
>  <opRet>0</opRet>
>  <opErrno>0</opErrno>
>  <opErrstr/>
>  <snapList>
>    <count>0</count>
>  </snapList>
></cliOutput>
>[kubeexec] ERROR 2018/06/08 08:45:50 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume stop vol_a4a6e4892da299f6c5634b8a2def697e force] on glusterfs-storage-vsh2m: Err[command terminated with exit code 1]: Stdout [Error : Request timed out
>]: Stderr []
>[cmdexec] ERROR 2018/06/08 08:45:50 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:144: Unable to stop volume vol_a4a6e4892da299f6c5634b8a2def697e: Unable to execute command on glusterfs-storage-vsh2m: 
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 325.131Âµs
>[kubeexec] DEBUG 2018/06/08 08:45:50 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:33 UTC; 2h 22min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48fed567_6ae4_11e8_bc4b_005056a5f18a.slice/docker-a506d9aaca4b440d184df40d01e90b2040811c8090bf4c19d892081307b28c0c.scope/system.slice/glusterd.service
>           ââ  433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1228 /usr/sbin/glusterfsd -s 10.70.46.187 --volfile-id heketidbstorage.10.70.46.187.var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.187-var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.pid -S /var/run/gluster/c08898c2e3b0190c62df3a9d4a013c92.socket --brick-name /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.log --xlator-option *-posix.glusterd-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ14272 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/8fb729a0bbd51b204491c603c84acaa4.socket --xlator-option *replicate*.node-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034
>
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Unit glusterd.service entered failed state.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: glusterd.service failed.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:33 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 08:45:50 Periodic health check status: node 278bd6b4e16a8e62ef15aaae22e6abc1 up=true
>[cmdexec] INFO 2018/06/08 08:45:50 Check Glusterd service status in node dhcp46-122.lab.eng.blr.redhat.com
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 4.141309ms
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 1.259317ms
>[kubeexec] ERROR 2018/06/08 08:45:50 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume delete vol_966a92ed3e4374a5e634f7b133c49e52] on glusterfs-storage-pg4xc: Err[command terminated with exit code 1]: Stdout []: Stderr [volume delete: vol_966a92ed3e4374a5e634f7b133c49e52: failed: Another transaction is in progress for vol_966a92ed3e4374a5e634f7b133c49e52. Please try again after sometime.
>]
>[cmdexec] ERROR 2018/06/08 08:45:50 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:153: Unable to delete volume vol_966a92ed3e4374a5e634f7b133c49e52: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_966a92ed3e4374a5e634f7b133c49e52: failed: Another transaction is in progress for vol_966a92ed3e4374a5e634f7b133c49e52. Please try again after sometime.
>[heketi] ERROR 2018/06/08 08:45:50 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:525: Unable to delete volume: Unable to delete volume vol_966a92ed3e4374a5e634f7b133c49e52: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_966a92ed3e4374a5e634f7b133c49e52: failed: Another transaction is in progress for vol_966a92ed3e4374a5e634f7b133c49e52. Please try again after sometime.
>[heketi] ERROR 2018/06/08 08:45:50 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:429: Error executing delete volume: Unable to delete volume vol_966a92ed3e4374a5e634f7b133c49e52: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_966a92ed3e4374a5e634f7b133c49e52: failed: Another transaction is in progress for vol_966a92ed3e4374a5e634f7b133c49e52. Please try again after sometime.
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 1.67263ms
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[asynchttp] INFO 2018/06/08 08:45:50 asynchttp.go:292: Completed job f57802d82a15df4af9a77cc5ba0a7516 in 2m0.912541295s
>[heketi] ERROR 2018/06/08 08:45:50 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:1185: Delete Volume Failed: Unable to delete volume vol_966a92ed3e4374a5e634f7b133c49e52: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_966a92ed3e4374a5e634f7b133c49e52: failed: Another transaction is in progress for vol_966a92ed3e4374a5e634f7b133c49e52. Please try again after sometime.
>[negroni] Completed 200 OK in 5.005378ms
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 2.61157ms
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 985.012Âµs
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 1.843781ms
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 1.16153ms
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 176.224Âµs
>[kubeexec] ERROR 2018/06/08 08:45:50 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume delete vol_a4a6e4892da299f6c5634b8a2def697e] on glusterfs-storage-vsh2m: Err[command terminated with exit code 1]: Stdout []: Stderr [volume delete: vol_a4a6e4892da299f6c5634b8a2def697e: failed: Another transaction is in progress for vol_a4a6e4892da299f6c5634b8a2def697e. Please try again after sometime.
>]
>[cmdexec] ERROR 2018/06/08 08:45:50 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:153: Unable to delete volume vol_a4a6e4892da299f6c5634b8a2def697e: Unable to execute command on glusterfs-storage-vsh2m: volume delete: vol_a4a6e4892da299f6c5634b8a2def697e: failed: Another transaction is in progress for vol_a4a6e4892da299f6c5634b8a2def697e. Please try again after sometime.
>[heketi] ERROR 2018/06/08 08:45:50 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:525: Unable to delete volume: Unable to delete volume vol_a4a6e4892da299f6c5634b8a2def697e: Unable to execute command on glusterfs-storage-vsh2m: volume delete: vol_a4a6e4892da299f6c5634b8a2def697e: failed: Another transaction is in progress for vol_a4a6e4892da299f6c5634b8a2def697e. Please try again after sometime.
>[heketi] ERROR 2018/06/08 08:45:50 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:429: Error executing delete volume: Unable to delete volume vol_a4a6e4892da299f6c5634b8a2def697e: Unable to execute command on glusterfs-storage-vsh2m: volume delete: vol_a4a6e4892da299f6c5634b8a2def697e: failed: Another transaction is in progress for vol_a4a6e4892da299f6c5634b8a2def697e. Please try again after sometime.
>[heketi] ERROR 2018/06/08 08:45:50 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:1185: Delete Volume Failed: Unable to delete volume vol_a4a6e4892da299f6c5634b8a2def697e: Unable to execute command on glusterfs-storage-vsh2m: volume delete: vol_a4a6e4892da299f6c5634b8a2def697e: failed: Another transaction is in progress for vol_a4a6e4892da299f6c5634b8a2def697e. Please try again after sometime.
>[asynchttp] INFO 2018/06/08 08:45:50 asynchttp.go:292: Completed job 9e1718475edb952ba52dafee886587f8 in 2m0.812423196s
>[negroni] Started GET /queue/f57802d82a15df4af9a77cc5ba0a7516
>[negroni] Completed 500 Internal Server Error in 269.925Âµs
>[negroni] Started GET /queue/9e1718475edb952ba52dafee886587f8
>[negroni] Completed 500 Internal Server Error in 240.441Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 168.267Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 200 OK in 254.601Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 4.647934ms
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 3.224815ms
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 900.977Âµs
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 1.561068ms
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 1.16295ms
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 1.051042ms
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 650.239Âµs
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 829.271Âµs
>[kubeexec] ERROR 2018/06/08 08:45:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume delete vol_9c85de66c12db0a72b4d16fe888ff74d] on glusterfs-storage-gxp7c: Err[command terminated with exit code 1]: Stdout [Error : Request timed out
>]: Stderr []
>[cmdexec] ERROR 2018/06/08 08:45:52 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:153: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: 
>[heketi] ERROR 2018/06/08 08:45:52 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:525: Unable to delete volume: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: 
>[heketi] ERROR 2018/06/08 08:45:52 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:429: Error executing delete volume: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: 
>[heketi] ERROR 2018/06/08 08:45:52 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:1185: Delete Volume Failed: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: 
>[asynchttp] INFO 2018/06/08 08:45:52 asynchttp.go:292: Completed job a9e6acc2efeee7febda192bc9c56b54e in 2m2.802710603s
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 152.63Âµs
>[negroni] Started GET /queue/a9e6acc2efeee7febda192bc9c56b54e
>[negroni] Completed 500 Internal Server Error in 202.237Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 5.031912ms
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 3.232661ms
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 1.065085ms
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 1.854986ms
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 1.402887ms
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 632.554Âµs
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 588.298Âµs
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 612.95Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 204.085Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 563.032Âµs
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 1.145084ms
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 649.686Âµs
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 1.014036ms
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 1.080402ms
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 1.029899ms
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 611.997Âµs
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 619.701Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 170.884Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 391.651Âµs
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 682.261Âµs
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 1.074255ms
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 1.016093ms
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 651.301Âµs
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 975.498Âµs
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 646.049Âµs
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 716.153Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 232.702Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 719.881Âµs
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 979.321Âµs
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 678.986Âµs
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 637.027Âµs
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 1.067369ms
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 680.466Âµs
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 607.672Âµs
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 867.089Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 146.905Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 244.924Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 565.429Âµs
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 1.084175ms
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 733.554Âµs
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 662.225Âµs
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 1.075006ms
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 613.926Âµs
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 609.461Âµs
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 1.019185ms
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 183.301Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 464.719Âµs
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 757.623Âµs
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 662.498Âµs
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 714.106Âµs
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 949.721Âµs
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 589.82Âµs
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 597.661Âµs
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 565.27Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 210.275Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 584.379Âµs
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 650.908Âµs
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 607.767Âµs
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 1.232287ms
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 739.407Âµs
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 1.084132ms
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 1.102279ms
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 1.018209ms
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 231.208Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 540.379Âµs
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 1.083562ms
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 1.451779ms
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 657.63Âµs
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 615.866Âµs
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 566.449Âµs
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 578.236Âµs
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 1.101095ms
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 127.067Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 376.235Âµs
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 1.027918ms
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 631.478Âµs
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 616.632Âµs
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 554.297Âµs
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 519.156Âµs
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 1.057909ms
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 902.002Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 222.302Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 362.74Âµs
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 883.981Âµs
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 1.072649ms
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 607.004Âµs
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 553.284Âµs
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 602.305Âµs
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 736.442Âµs
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 975.219Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 139.034Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 183.305Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 485.59Âµs
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 1.088546ms
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 986.039Âµs
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 604.102Âµs
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 596.673Âµs
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 635.7Âµs
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 910.684Âµs
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 631.467Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 236.552Âµs
>[negroni] Started DELETE /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Started DELETE /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Started DELETE /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 202 Accepted in 13.088735ms
>[asynchttp] INFO 2018/06/08 08:46:05 asynchttp.go:288: Started job b7642ac5823fb25c80d83504800b9c18
>[heketi] INFO 2018/06/08 08:46:05 Started async operation: Delete Volume
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 120.479Âµs
>[negroni] Completed 202 Accepted in 19.196566ms
>[asynchttp] INFO 2018/06/08 08:46:05 asynchttp.go:288: Started job 9abffd27d4b48c8f2f0e699632a69408
>[heketi] INFO 2018/06/08 08:46:05 Started async operation: Delete Volume
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 168.816Âµs
>[negroni] Completed 202 Accepted in 34.74259ms
>[asynchttp] INFO 2018/06/08 08:46:05 asynchttp.go:288: Started job c2b9e81466aab79d146584d4fe207629
>[heketi] INFO 2018/06/08 08:46:05 Started async operation: Delete Volume
>[negroni] Started GET /volumes
>[negroni] Started GET /queue/c2b9e81466aab79d146584d4fe207629
>[negroni] Completed 200 OK in 150.257Âµs
>[negroni] Completed 200 OK in 5.229066ms
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 2.30868ms
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 578.366Âµs
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 1.223012ms
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 766.437Âµs
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 1.00878ms
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 1.009005ms
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 938.057Âµs
>[kubeexec] DEBUG 2018/06/08 08:46:06 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: gluster --mode=script snapshot list vol_a4a6e4892da299f6c5634b8a2def697e --xml
>Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
><cliOutput>
>  <opRet>0</opRet>
>  <opErrno>0</opErrno>
>  <opErrstr/>
>  <snapList>
>    <count>0</count>
>  </snapList>
></cliOutput>
>[kubeexec] ERROR 2018/06/08 08:46:06 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume stop vol_a4a6e4892da299f6c5634b8a2def697e force] on glusterfs-storage-vsh2m: Err[command terminated with exit code 1]: Stdout []: Stderr [volume stop: vol_a4a6e4892da299f6c5634b8a2def697e: failed: Another transaction is in progress for vol_a4a6e4892da299f6c5634b8a2def697e. Please try again after sometime.
>]
>[cmdexec] ERROR 2018/06/08 08:46:06 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:144: Unable to stop volume vol_a4a6e4892da299f6c5634b8a2def697e: Unable to execute command on glusterfs-storage-vsh2m: volume stop: vol_a4a6e4892da299f6c5634b8a2def697e: failed: Another transaction is in progress for vol_a4a6e4892da299f6c5634b8a2def697e. Please try again after sometime.
>[kubeexec] ERROR 2018/06/08 08:46:06 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume delete vol_a4a6e4892da299f6c5634b8a2def697e] on glusterfs-storage-vsh2m: Err[command terminated with exit code 1]: Stdout []: Stderr [volume delete: vol_a4a6e4892da299f6c5634b8a2def697e: failed: Another transaction is in progress for vol_a4a6e4892da299f6c5634b8a2def697e. Please try again after sometime.
>]
>[cmdexec] ERROR 2018/06/08 08:46:06 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:153: Unable to delete volume vol_a4a6e4892da299f6c5634b8a2def697e: Unable to execute command on glusterfs-storage-vsh2m: volume delete: vol_a4a6e4892da299f6c5634b8a2def697e: failed: Another transaction is in progress for vol_a4a6e4892da299f6c5634b8a2def697e. Please try again after sometime.
>[heketi] ERROR 2018/06/08 08:46:06 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:525: Unable to delete volume: Unable to delete volume vol_a4a6e4892da299f6c5634b8a2def697e: Unable to execute command on glusterfs-storage-vsh2m: volume delete: vol_a4a6e4892da299f6c5634b8a2def697e: failed: Another transaction is in progress for vol_a4a6e4892da299f6c5634b8a2def697e. Please try again after sometime.
>[heketi] ERROR 2018/06/08 08:46:06 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:429: Error executing delete volume: Unable to delete volume vol_a4a6e4892da299f6c5634b8a2def697e: Unable to execute command on glusterfs-storage-vsh2m: volume delete: vol_a4a6e4892da299f6c5634b8a2def697e: failed: Another transaction is in progress for vol_a4a6e4892da299f6c5634b8a2def697e. Please try again after sometime.
>[heketi] ERROR 2018/06/08 08:46:06 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:1185: Delete Volume Failed: Unable to delete volume vol_a4a6e4892da299f6c5634b8a2def697e: Unable to execute command on glusterfs-storage-vsh2m: volume delete: vol_a4a6e4892da299f6c5634b8a2def697e: failed: Another transaction is in progress for vol_a4a6e4892da299f6c5634b8a2def697e. Please try again after sometime.
>[asynchttp] INFO 2018/06/08 08:46:06 asynchttp.go:292: Completed job c2b9e81466aab79d146584d4fe207629 in 734.184107ms
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 261.631Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 208.383Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 236.031Âµs
>[negroni] Started GET /queue/c2b9e81466aab79d146584d4fe207629
>[negroni] Completed 500 Internal Server Error in 164.424Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 4.683236ms
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 2.317349ms
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 617.533Âµs
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 1.443724ms
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 1.162033ms
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 638.136Âµs
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 616.874Âµs
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 704.529Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 213.59Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 186.308Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 106.156Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 839.256Âµs
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 1.195678ms
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 574.715Âµs
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 591.762Âµs
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 564.713Âµs
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 1.022235ms
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 625.741Âµs
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 1.050923ms
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 190.028Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 252.135Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 179.351Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 831.31Âµs
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 703.385Âµs
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 1.017908ms
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 1.002099ms
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 566.486Âµs
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 1.101944ms
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 939.489Âµs
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 575.798Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 253.502Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 144.905Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 152.279Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 571.554Âµs
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 1.035099ms
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 1.020013ms
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 560.316Âµs
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 952.716Âµs
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 946.496Âµs
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 624.052Âµs
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 965.609Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 152.505Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 132.285Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 146.692Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 669.83Âµs
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 778.46Âµs
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 567.097Âµs
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 633.68Âµs
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 766.004Âµs
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 552.794Âµs
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 516.722Âµs
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 544.041Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 119.93Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 129.471Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 152.885Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 160.691Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 569.031Âµs
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 639.385Âµs
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 641.486Âµs
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 703.061Âµs
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 574.185Âµs
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 567.411Âµs
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 550.058Âµs
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 610.105Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 127.713Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 131.18Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 105.289Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 479.773Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 95.276Âµs
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 641.926Âµs
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 628.738Âµs
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 555.063Âµs
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 538.253Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 84.583Âµs
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 509.925Âµs
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 553.636Âµs
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 494.643Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 183.271Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 159.222Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 131.675Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 794.461Âµs
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 1.090269ms
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 1.018193ms
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 568.112Âµs
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 628.784Âµs
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 603.919Âµs
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 548.853Âµs
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 509.522Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 144.205Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 163.576Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 93.335Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 565.676Âµs
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 639.807Âµs
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 680.529Âµs
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 566.983Âµs
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 525.289Âµs
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 621.788Âµs
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 969.106Âµs
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 1.067702ms
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 247.362Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 150.585Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 129.397Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 575.053Âµs
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 670.526Âµs
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 641.303Âµs
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 594.072Âµs
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 961.946Âµs
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 961.273Âµs
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 961.86Âµs
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 937.087Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 191.879Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 191.414Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 171.102Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 911.279Âµs
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 1.140928ms
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 554.184Âµs
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 574.557Âµs
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 613.75Âµs
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 519.888Âµs
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 637.361Âµs
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 984.999Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 236.689Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 180.336Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 210.915Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 819.378Âµs
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 686.43Âµs
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 652.871Âµs
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 741.787Âµs
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 583.776Âµs
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 559.237Âµs
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 1.602027ms
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 602.945Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 194.425Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 139.839Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 188.791Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 193.535Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 844.984Âµs
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 985.653Âµs
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 1.031582ms
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 951.803Âµs
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 998.125Âµs
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 957.119Âµs
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 949.097Âµs
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 941.996Âµs
>[negroni] Started DELETE /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 202 Accepted in 12.029817ms
>[asynchttp] INFO 2018/06/08 08:46:20 asynchttp.go:288: Started job 6c21d7f239114dec28a40bfe0657a19c
>[heketi] INFO 2018/06/08 08:46:20 Started async operation: Delete Volume
>[negroni] Started GET /queue/6c21d7f239114dec28a40bfe0657a19c
>[negroni] Completed 200 OK in 145.409Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 153.185Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 131.789Âµs
>[kubeexec] DEBUG 2018/06/08 08:46:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: gluster --mode=script snapshot list vol_a4a6e4892da299f6c5634b8a2def697e --xml
>Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
><cliOutput>
>  <opRet>0</opRet>
>  <opErrno>0</opErrno>
>  <opErrstr/>
>  <snapList>
>    <count>0</count>
>  </snapList>
></cliOutput>
>[kubeexec] ERROR 2018/06/08 08:46:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume stop vol_a4a6e4892da299f6c5634b8a2def697e force] on glusterfs-storage-vsh2m: Err[command terminated with exit code 1]: Stdout []: Stderr [volume stop: vol_a4a6e4892da299f6c5634b8a2def697e: failed: Another transaction is in progress for vol_a4a6e4892da299f6c5634b8a2def697e. Please try again after sometime.
>]
>[cmdexec] ERROR 2018/06/08 08:46:21 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:144: Unable to stop volume vol_a4a6e4892da299f6c5634b8a2def697e: Unable to execute command on glusterfs-storage-vsh2m: volume stop: vol_a4a6e4892da299f6c5634b8a2def697e: failed: Another transaction is in progress for vol_a4a6e4892da299f6c5634b8a2def697e. Please try again after sometime.
>[kubeexec] ERROR 2018/06/08 08:46:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume delete vol_a4a6e4892da299f6c5634b8a2def697e] on glusterfs-storage-vsh2m: Err[command terminated with exit code 1]: Stdout []: Stderr [volume delete: vol_a4a6e4892da299f6c5634b8a2def697e: failed: Another transaction is in progress for vol_a4a6e4892da299f6c5634b8a2def697e. Please try again after sometime.
>]
>[cmdexec] ERROR 2018/06/08 08:46:21 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:153: Unable to delete volume vol_a4a6e4892da299f6c5634b8a2def697e: Unable to execute command on glusterfs-storage-vsh2m: volume delete: vol_a4a6e4892da299f6c5634b8a2def697e: failed: Another transaction is in progress for vol_a4a6e4892da299f6c5634b8a2def697e. Please try again after sometime.
>[heketi] ERROR 2018/06/08 08:46:21 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:525: Unable to delete volume: Unable to delete volume vol_a4a6e4892da299f6c5634b8a2def697e: Unable to execute command on glusterfs-storage-vsh2m: volume delete: vol_a4a6e4892da299f6c5634b8a2def697e: failed: Another transaction is in progress for vol_a4a6e4892da299f6c5634b8a2def697e. Please try again after sometime.
>[heketi] ERROR 2018/06/08 08:46:21 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:429: Error executing delete volume: Unable to delete volume vol_a4a6e4892da299f6c5634b8a2def697e: Unable to execute command on glusterfs-storage-vsh2m: volume delete: vol_a4a6e4892da299f6c5634b8a2def697e: failed: Another transaction is in progress for vol_a4a6e4892da299f6c5634b8a2def697e. Please try again after sometime.
>[heketi] ERROR 2018/06/08 08:46:21 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:1185: Delete Volume Failed: Unable to delete volume vol_a4a6e4892da299f6c5634b8a2def697e: Unable to execute command on glusterfs-storage-vsh2m: volume delete: vol_a4a6e4892da299f6c5634b8a2def697e: failed: Another transaction is in progress for vol_a4a6e4892da299f6c5634b8a2def697e. Please try again after sometime.
>[asynchttp] INFO 2018/06/08 08:46:21 asynchttp.go:292: Completed job 6c21d7f239114dec28a40bfe0657a19c in 765.629949ms
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 185.697Âµs
>[negroni] Started GET /queue/6c21d7f239114dec28a40bfe0657a19c
>[negroni] Completed 500 Internal Server Error in 191.963Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 187.633Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 137.899Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 4.393522ms
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 2.420794ms
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 1.039195ms
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 1.213887ms
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 1.594482ms
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 587.059Âµs
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 555.121Âµs
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 686.892Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 221.374Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 225.385Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 118.653Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 508.506Âµs
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 674.882Âµs
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 578.674Âµs
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 613.37Âµs
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 529.405Âµs
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 554.05Âµs
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 503.13Âµs
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 618.491Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 229.668Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 253.145Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 197.751Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 563.918Âµs
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 620.833Âµs
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 593.499Âµs
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 739.114Âµs
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 532.603Âµs
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 613.599Âµs
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 620.727Âµs
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 530.132Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 292.614Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 238.508Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 144.052Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 591.761Âµs
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 626.99Âµs
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 497.951Âµs
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 533.577Âµs
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 531.088Âµs
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 529.725Âµs
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 548.326Âµs
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 526.452Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 110.815Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 122.874Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 120.007Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 778.854Âµs
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 1.126481ms
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 1.122432ms
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 1.002803ms
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 1.022679ms
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 953.003Âµs
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 950.043Âµs
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 956.382Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 139.058Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 199.025Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 189.104Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 787.87Âµs
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 918.316Âµs
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 677.649Âµs
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 657.037Âµs
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 811.872Âµs
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 1.222931ms
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 989.828Âµs
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 548.559Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 140.615Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 130.509Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 91.266Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 127.641Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 850.557Âµs
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 1.076632ms
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 1.080738ms
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 935.134Âµs
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 676.564Âµs
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 760.419Âµs
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 592.732Âµs
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 535.488Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 214.651Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 116.652Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 142.83Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 242.805Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 150.166Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 772.214Âµs
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 715.584Âµs
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 1.035412ms
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 605.39Âµs
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 953.412Âµs
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 948.4Âµs
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 605.528Âµs
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 592.063Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 231.658Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 175.761Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 139.323Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 590.305Âµs
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 794.747Âµs
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 1.105515ms
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 1.037376ms
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 960.283Âµs
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 591.601Âµs
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 953.029Âµs
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 599.758Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 122.716Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 116.703Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 121.385Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 521.11Âµs
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 650.913Âµs
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 662.305Âµs
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 549.688Âµs
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 585.502Âµs
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 1.036012ms
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 522.165Âµs
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 942.88Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 196.881Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 212.604Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 140.126Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 800.444Âµs
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 1.036529ms
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 768.725Âµs
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 734.946Âµs
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 964.186Âµs
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 634.307Âµs
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 618.777Âµs
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 559.525Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 212.058Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 242.422Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 182.461Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 816.337Âµs
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 1.056432ms
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 661.439Âµs
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 869.24Âµs
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 659.745Âµs
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 598.067Âµs
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 609.744Âµs
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 903.917Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 196.301Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 236.199Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 140.206Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 521.13Âµs
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 707.604Âµs
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 626.534Âµs
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 531.411Âµs
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 678.454Âµs
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 806.589Âµs
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 547.144Âµs
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 985.356Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 78.633Âµs
>[negroni] Started DELETE /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 202 Accepted in 14.365264ms
>[asynchttp] INFO 2018/06/08 08:46:35 asynchttp.go:288: Started job ad59b27cce2d3a9ca1ce2455f01a9e56
>[heketi] INFO 2018/06/08 08:46:35 Started async operation: Delete Volume
>[negroni] Started GET /queue/ad59b27cce2d3a9ca1ce2455f01a9e56
>[negroni] Completed 200 OK in 113.935Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 205.086Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 135.553Âµs
>[kubeexec] DEBUG 2018/06/08 08:46:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: gluster --mode=script snapshot list vol_a4a6e4892da299f6c5634b8a2def697e --xml
>Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
><cliOutput>
>  <opRet>0</opRet>
>  <opErrno>0</opErrno>
>  <opErrstr/>
>  <snapList>
>    <count>0</count>
>  </snapList>
></cliOutput>
>[kubeexec] ERROR 2018/06/08 08:46:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume stop vol_a4a6e4892da299f6c5634b8a2def697e force] on glusterfs-storage-vsh2m: Err[command terminated with exit code 1]: Stdout []: Stderr [volume stop: vol_a4a6e4892da299f6c5634b8a2def697e: failed: Another transaction is in progress for vol_a4a6e4892da299f6c5634b8a2def697e. Please try again after sometime.
>]
>[cmdexec] ERROR 2018/06/08 08:46:36 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:144: Unable to stop volume vol_a4a6e4892da299f6c5634b8a2def697e: Unable to execute command on glusterfs-storage-vsh2m: volume stop: vol_a4a6e4892da299f6c5634b8a2def697e: failed: Another transaction is in progress for vol_a4a6e4892da299f6c5634b8a2def697e. Please try again after sometime.
>[kubeexec] ERROR 2018/06/08 08:46:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume delete vol_a4a6e4892da299f6c5634b8a2def697e] on glusterfs-storage-vsh2m: Err[command terminated with exit code 1]: Stdout []: Stderr [volume delete: vol_a4a6e4892da299f6c5634b8a2def697e: failed: Another transaction is in progress for vol_a4a6e4892da299f6c5634b8a2def697e. Please try again after sometime.
>]
>[cmdexec] ERROR 2018/06/08 08:46:36 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:153: Unable to delete volume vol_a4a6e4892da299f6c5634b8a2def697e: Unable to execute command on glusterfs-storage-vsh2m: volume delete: vol_a4a6e4892da299f6c5634b8a2def697e: failed: Another transaction is in progress for vol_a4a6e4892da299f6c5634b8a2def697e. Please try again after sometime.
>[heketi] ERROR 2018/06/08 08:46:36 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:525: Unable to delete volume: Unable to delete volume vol_a4a6e4892da299f6c5634b8a2def697e: Unable to execute command on glusterfs-storage-vsh2m: volume delete: vol_a4a6e4892da299f6c5634b8a2def697e: failed: Another transaction is in progress for vol_a4a6e4892da299f6c5634b8a2def697e. Please try again after sometime.
>[heketi] ERROR 2018/06/08 08:46:36 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:429: Error executing delete volume: Unable to delete volume vol_a4a6e4892da299f6c5634b8a2def697e: Unable to execute command on glusterfs-storage-vsh2m: volume delete: vol_a4a6e4892da299f6c5634b8a2def697e: failed: Another transaction is in progress for vol_a4a6e4892da299f6c5634b8a2def697e. Please try again after sometime.
>[asynchttp] INFO 2018/06/08 08:46:36 asynchttp.go:292: Completed job ad59b27cce2d3a9ca1ce2455f01a9e56 in 716.649398ms
>[heketi] ERROR 2018/06/08 08:46:36 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:1185: Delete Volume Failed: Unable to delete volume vol_a4a6e4892da299f6c5634b8a2def697e: Unable to execute command on glusterfs-storage-vsh2m: volume delete: vol_a4a6e4892da299f6c5634b8a2def697e: failed: Another transaction is in progress for vol_a4a6e4892da299f6c5634b8a2def697e. Please try again after sometime.
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 201.839Âµs
>[negroni] Started GET /queue/ad59b27cce2d3a9ca1ce2455f01a9e56
>[negroni] Completed 500 Internal Server Error in 238.028Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 4.459436ms
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 2.778976ms
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 712.247Âµs
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 1.907219ms
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 1.229608ms
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 975.772Âµs
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 579.053Âµs
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 595.698Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 104.455Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 153.865Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 204.045Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 137.959Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 140.31Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 520.563Âµs
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 645.413Âµs
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 715.066Âµs
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 561.328Âµs
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 625.166Âµs
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 560.46Âµs
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 536.027Âµs
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 494.003Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 245.775Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 192.198Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 134.332Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 1.054482ms
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 752.544Âµs
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 1.016739ms
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 621.62Âµs
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 706.491Âµs
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 631.691Âµs
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 622.07Âµs
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 537.1Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 195.045Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 221.352Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 130.272Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 545.102Âµs
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 2.597689ms
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 1.558791ms
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 1.018081ms
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 985.408Âµs
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 612.995Âµs
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 579.937Âµs
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 642.539Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 196.009Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 115.736Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 89.622Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 783.967Âµs
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 1.100204ms
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 1.035712ms
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 979.463Âµs
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 1.016688ms
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 1.711904ms
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 1.004307ms
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 692.348Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 204.136Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 168.757Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 135.622Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 828.197Âµs
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 668.475Âµs
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 696.948Âµs
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 1.040629ms
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 592.382Âµs
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 1.00385ms
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 1.000446ms
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 966.282Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 208.848Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 441.09Âµs
>[negroni] Completed 200 OK in 153.759Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 177.056Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 592.912Âµs
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 684.186Âµs
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 630.346Âµs
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 593.171Âµs
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 620.421Âµs
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 579.422Âµs
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 1.073398ms
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 593.293Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 204.201Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 179.435Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 164.678Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 688.452Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 133.787Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 99.129Âµs
>[negroni] Completed 200 OK in 1.118348ms
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 628.109Âµs
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 973.446Âµs
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 578.484Âµs
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 953.863Âµs
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 560.39Âµs
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 1.032939ms
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 190.501Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 206.663Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 122.283Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 494.264Âµs
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 944.252Âµs
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 1.027796ms
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 1.012799ms
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 613.106Âµs
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 544.636Âµs
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 605.065Âµs
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 1.008816ms
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 169.563Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 177.981Âµs
>[negroni] Completed 200 OK in 98.851Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 806.861Âµs
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 1.033725ms
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 1.061169ms
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 1.038749ms
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 579.513Âµs
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 1.014536ms
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 542.434Âµs
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 1.690267ms
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 207.435Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 212.891Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 130.459Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 826.024Âµs
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 1.022265ms
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 993.483Âµs
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 675.877Âµs
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 597.785Âµs
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 613.416Âµs
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 1.043653ms
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 858.8Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 176.066Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 127.034Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 444.92Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 495.313Âµs
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 1.058315ms
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 661.788Âµs
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 587.891Âµs
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 1.035465ms
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 560.356Âµs
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 1.018355ms
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 574.644Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 113.253Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 156.013Âµs
>[negroni] Completed 200 OK in 82.259Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 631.316Âµs
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 785.149Âµs
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 1.034606ms
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 998.099Âµs
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 988.449Âµs
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 1.135268ms
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 1.125204ms
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 996.376Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 135.389Âµs
>[negroni] Started DELETE /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 202 Accepted in 14.017764ms
>[asynchttp] INFO 2018/06/08 08:46:50 asynchttp.go:288: Started job 548c6fbe203bb93f9a4ecb56f976926d
>[heketi] INFO 2018/06/08 08:46:50 Started async operation: Delete Volume
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 123.606Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 145.387Âµs
>[negroni] Completed 200 OK in 243.122Âµs
>[kubeexec] DEBUG 2018/06/08 08:46:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: gluster --mode=script snapshot list vol_a4a6e4892da299f6c5634b8a2def697e --xml
>Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
><cliOutput>
>  <opRet>0</opRet>
>  <opErrno>0</opErrno>
>  <opErrstr/>
>  <snapList>
>    <count>0</count>
>  </snapList>
></cliOutput>
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 157.171Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 4.074252ms
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 2.334264ms
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 590.107Âµs
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 1.045169ms
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 560.226Âµs
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 566.836Âµs
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 611.696Âµs
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 584.324Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 136.809Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 172.255Âµs
>[negroni] Completed 200 OK in 197.878Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 183.049Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 114.82Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 165.376Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 135.815Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 649.55Âµs
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 1.129071ms
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 589.095Âµs
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 607.804Âµs
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 505.81Âµs
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 511.676Âµs
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 1.008542ms
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 993.062Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 124.091Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 196.935Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 287.464Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 124.552Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 940.132Âµs
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 1.029012ms
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 595.439Âµs
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 1.046699ms
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 707.687Âµs
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 609.315Âµs
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 582.529Âµs
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 515.405Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 207.764Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 155.378Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 209.335Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 130.996Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 970Âµs
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 671.03Âµs
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 596.38Âµs
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 985.176Âµs
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 1.002518ms
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 954.732Âµs
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 1.054805ms
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 1.025096ms
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 195.848Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 121.995Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 116.328Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 55.359Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 932.087Âµs
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 1.575072ms
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 1.131005ms
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 1.196014ms
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 1.061629ms
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 1.038972ms
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 686.215Âµs
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 1.017275ms
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 182.286Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 204.726Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 191.665Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 161.276Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 891.027Âµs
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 1.091652ms
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 987.96Âµs
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 1.079111ms
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 1.611038ms
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 1.012445ms
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 980.035Âµs
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 1.125475ms
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 188.959Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 132.409Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 134.167Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 153.955Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 575.637Âµs
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 635.953Âµs
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 955.728Âµs
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 1.063432ms
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 566.674Âµs
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 995.443Âµs
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 988.623Âµs
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 1.023989ms
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 165.759Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 187.588Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 148.757Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 150.096Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 177.758Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 219.395Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 1.022355ms
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 1.690588ms
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 1.119769ms
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 1.083805ms
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 1.079512ms
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 577.963Âµs
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 1.058226ms
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 188.476Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 105.012Âµs
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 1.071898ms
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 195.469Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 199.445Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 246.548Âµs
>[negroni] Completed 200 OK in 247.528Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 605.927Âµs
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 707.347Âµs
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 823.272Âµs
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 1.013833ms
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 911.46Âµs
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 1.085694ms
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 1.00098ms
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 548.553Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 142.299Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 122.103Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 109.32Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 188.409Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 986.823Âµs
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 614.042Âµs
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 1.023169ms
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 600.265Âµs
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 522.514Âµs
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 582.002Âµs
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 1.031765ms
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 614.907Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 200.152Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 178.166Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 173.513Âµs
>[negroni] Completed 200 OK in 272.181Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 634.259Âµs
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 1.536631ms
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 657.378Âµs
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 1.017186ms
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 1.030635ms
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 983.263Âµs
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 984.436Âµs
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 589.818Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 193.162Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 165.686Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 121.406Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 287.75Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 601.371Âµs
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 747.772Âµs
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 651.729Âµs
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 623.863Âµs
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 1.058888ms
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 573.392Âµs
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 624.583Âµs
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 618.316Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 133.171Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 168.152Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 289.295Âµs
>[negroni] Completed 200 OK in 96.065Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 573.486Âµs
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 1.087331ms
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 1.120225ms
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 1.117959ms
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 1.023028ms
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 643.242Âµs
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 760.283Âµs
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 593.113Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 260.135Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 257.695Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 231.708Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 98.785Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 148.487Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 908.349Âµs
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 700.626Âµs
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 617.964Âµs
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 603.393Âµs
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 1.043849ms
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 548.994Âµs
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 187.449Âµs
>[negroni] Completed 200 OK in 1.532405ms
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 1.001869ms
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 188.275Âµs
>[negroni] Completed 200 OK in 142.029Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 189.512Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 246.128Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 152.065Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 130.515Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 601.656Âµs
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 666.892Âµs
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 1.076178ms
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 541.041Âµs
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 586.499Âµs
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 575.669Âµs
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 555.516Âµs
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 570.495Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 206.245Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 267.678Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 210.529Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 232.925Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 960.917Âµs
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 1.018895ms
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 858.291Âµs
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 1.110798ms
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 934.722Âµs
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 919.39Âµs
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 583.319Âµs
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 608.084Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 181.933Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 229.155Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 181.906Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 146.363Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 951.149Âµs
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 585.166Âµs
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 625.603Âµs
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 591.757Âµs
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 625.258Âµs
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 599.178Âµs
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 688.404Âµs
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 528.94Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 130.839Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 224.551Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 159.868Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 136.606Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 1.167088ms
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 1.341673ms
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 826.7Âµs
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 629.493Âµs
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 901.638Âµs
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 639.394Âµs
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 547.664Âµs
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 742.775Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 156.999Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 192.211Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 130.168Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 100.192Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 583.25Âµs
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 1.113628ms
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 948.374Âµs
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 807.086Âµs
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 574.664Âµs
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 526.422Âµs
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 593.58Âµs
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 556.067Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 138.548Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 132.733Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 147.323Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 177.392Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 636.23Âµs
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 822.224Âµs
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 624.957Âµs
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 602.352Âµs
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 575.373Âµs
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 1.033009ms
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 573.229Âµs
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 988.983Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 178.162Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 282.961Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 308.864Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 198.935Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 214.345Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 1.547762ms
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 674.327Âµs
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 621.436Âµs
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 614.415Âµs
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 689.122Âµs
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 599.95Âµs
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 936.493Âµs
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 866.446Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 124.599Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 167.595Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 167.016Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 153.423Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 186.232Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 286.638Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 122.819Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 539.57Âµs
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 1.152894ms
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 995.629Âµs
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 952.6Âµs
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 1.060329ms
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 909.673Âµs
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 1.002477ms
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 624.378Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 145.291Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 186.385Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 133.983Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 83.928Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 595.215Âµs
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 772.008Âµs
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 732.897Âµs
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 988.355Âµs
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 553.462Âµs
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 750.47Âµs
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 975.295Âµs
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 723.548Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 191.518Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 160.55Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 292.325Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 293.752Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 918.31Âµs
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 1.111338ms
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 978.763Âµs
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 677.636Âµs
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 1.038005ms
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 998.216Âµs
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 650.317Âµs
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 910.833Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 128.056Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 227.408Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 187.668Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 111.823Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 594.356Âµs
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 635.952Âµs
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 600.412Âµs
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 578.914Âµs
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 540.307Âµs
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 577.436Âµs
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 553.017Âµs
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 619.372Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 145.053Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 141.731Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 145.273Âµs
>[negroni] Completed 200 OK in 212.383Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 930.855Âµs
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 1.193394ms
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 960.853Âµs
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 1.072796ms
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 1.03631ms
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 605.745Âµs
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 580.157Âµs
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 986.715Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 202.885Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 255.128Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 156.256Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 114.321Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 610.632Âµs
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 788.599Âµs
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 710.807Âµs
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 879.589Âµs
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 957.052Âµs
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 543.043Âµs
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 892.117Âµs
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 615.927Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 112.1Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 313.161Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 128.542Âµs
>[negroni] Completed 200 OK in 198.501Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 149.833Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 128.73Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 1.225188ms
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 604.544Âµs
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 641.046Âµs
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 559.161Âµs
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 537.842Âµs
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 573.403Âµs
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 619.807Âµs
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 571.319Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 122.877Âµs
>[negroni] Completed 200 OK in 176.504Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 232.551Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 152.781Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 123.417Âµs
>[negroni] Completed 200 OK in 135.644Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 942.669Âµs
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 1.082846ms
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 1.048012ms
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 957.723Âµs
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 829.56Âµs
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 949.333Âµs
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 545.149Âµs
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 1.052442ms
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 190.748Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 184.488Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 194.849Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 377.146Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 959.755Âµs
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 1.142902ms
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 648.118Âµs
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 692.59Âµs
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 555.376Âµs
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 1.006539ms
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 1.027479ms
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 1.038825ms
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 197.265Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 195.525Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 108.535Âµs
>[negroni] Completed 200 OK in 178.496Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 820.154Âµs
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 1.340093ms
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 674.656Âµs
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 838.119Âµs
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 620.554Âµs
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 731.268Âµs
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 628.427Âµs
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 589.119Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 190.886Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 190.785Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 186.319Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 202.645Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 1.398007ms
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 885.105Âµs
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 796.785Âµs
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 988.689Âµs
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 773.696Âµs
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 951.801Âµs
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 565.966Âµs
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 555.261Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 162.508Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 188.709Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 152.66Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 96.743Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 594.557Âµs
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 1.018422ms
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 820.029Âµs
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 629.595Âµs
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 611.47Âµs
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 757.565Âµs
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 566.483Âµs
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 531.38Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 162.568Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 131.375Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 153.663Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 118.513Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 181.139Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 1.028332ms
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 637.757Âµs
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 1.035215ms
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 975.26Âµs
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 588.818Âµs
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 578.038Âµs
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 860.668Âµs
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 1.199304ms
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 131.156Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 173.059Âµs
>[negroni] Completed 200 OK in 144.218Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 161.361Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 230.785Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 899.853Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 212.821Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 146.892Âµs
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 592.65Âµs
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 648.584Âµs
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 595.479Âµs
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 593.516Âµs
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 564.267Âµs
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 922.334Âµs
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 1.037502ms
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 244.598Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 147.176Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 172.372Âµs
>[negroni] Completed 200 OK in 219.834Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 630.548Âµs
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 1.075931ms
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 609.092Âµs
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 545.226Âµs
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 580.902Âµs
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 573.239Âµs
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 521.45Âµs
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 565.484Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 279.698Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 139.093Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 254.161Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 157.349Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 1.016523ms
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 656.733Âµs
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 611.145Âµs
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 1.005402ms
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 970.642Âµs
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 532.566Âµs
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 991.753Âµs
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 988.363Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 182.449Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 197.599Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 196.114Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 448.059Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 646.854Âµs
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 1.061232ms
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 1.022572ms
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 539.342Âµs
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 598.678Âµs
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 798.267Âµs
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 534.665Âµs
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 533.657Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 157.197Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 150.021Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 139.954Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 358.441Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 966.076Âµs
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 1.053158ms
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 957.79Âµs
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 1.223814ms
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 965.129Âµs
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 1.057972ms
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 973.756Âµs
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 740.081Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 180.586Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 140.305Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 110.99Âµs
>[negroni] Completed 200 OK in 158.665Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 559.651Âµs
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 645.258Âµs
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 513.697Âµs
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 634.103Âµs
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 514.885Âµs
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 619.645Âµs
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 961.826Âµs
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 951.587Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 187.228Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 140.665Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 134.574Âµs
>[negroni] Completed 200 OK in 313.565Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 201.282Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 1.059888ms
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 119.02Âµs
>[negroni] Completed 200 OK in 1.536351ms
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 615.673Âµs
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 991.833Âµs
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 654.782Âµs
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 1.259748ms
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 528.994Âµs
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 1.026582ms
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 148.815Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 88.149Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 208.561Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 254.208Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 297.567Âµs
>[negroni] Completed 200 OK in 137.026Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 1.043362ms
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 1.073615ms
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 1.083085ms
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 781.405Âµs
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 647.744Âµs
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 1.057475ms
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 766.82Âµs
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 947.493Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 217.512Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 197.166Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 217.942Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 209.486Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 1.054129ms
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 993.853Âµs
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 764.055Âµs
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 994.532Âµs
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 995.097Âµs
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 590.653Âµs
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 784.783Âµs
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 575.773Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 120.51Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 132.643Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 159.812Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 151.022Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 668.485Âµs
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 1.049398ms
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 615.663Âµs
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 974.546Âµs
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 982.29Âµs
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 1.396029ms
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 564.373Âµs
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 971.953Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 175.51Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 171.712Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 172.139Âµs
>[negroni] Completed 200 OK in 275.934Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 944.65Âµs
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 1.021462ms
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 968.059Âµs
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 553.671Âµs
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 580.87Âµs
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 709.094Âµs
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 600.409Âµs
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 1.204791ms
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 135.547Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 189.025Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 188.786Âµs
>[negroni] Completed 200 OK in 284.79Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 590.67Âµs
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 623.875Âµs
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 624.437Âµs
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 561.732Âµs
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 980.68Âµs
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 578.035Âµs
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 921.916Âµs
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 555.12Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 126.14Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 120.359Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 164.22Âµs
>[negroni] Completed 200 OK in 98.449Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 948.527Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 188.455Âµs
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 962.936Âµs
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 968.223Âµs
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 1.045225ms
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 969.115Âµs
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 817.136Âµs
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 1.082758ms
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 993.672Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 166.006Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 187.721Âµs
>[negroni] Completed 200 OK in 99.309Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 180.313Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 387.927Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 830.415Âµs
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 612.448Âµs
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 555.507Âµs
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 537.881Âµs
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 584.956Âµs
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 563.405Âµs
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 535.088Âµs
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 498.526Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 137.123Âµs
>[negroni] Completed 200 OK in 209.687Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 192.998Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 149.239Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 132.602Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 147.169Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 1.429823ms
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 1.012139ms
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 1.043396ms
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 727.016Âµs
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 1.040225ms
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 1.098388ms
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 678.121Âµs
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 1.000999ms
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 173.785Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 234.172Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 175.889Âµs
>[negroni] Completed 200 OK in 132.332Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 764.317Âµs
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 812.241Âµs
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 644.092Âµs
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 1.74163ms
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 617.545Âµs
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 734.786Âµs
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 785.52Âµs
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 1.542711ms
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 280.684Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 191.819Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 137.141Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 103.843Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 566.158Âµs
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 775.7Âµs
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 669.008Âµs
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 1.087582ms
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 662.112Âµs
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 672.561Âµs
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 504.184Âµs
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 546.008Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 173.452Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 132.263Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 150.773Âµs
>[negroni] Completed 200 OK in 221.101Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 992.382Âµs
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 685.635Âµs
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 1.064846ms
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 1.037368ms
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 949.833Âµs
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 759.914Âµs
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 1.514373ms
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 1.913129ms
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 200 OK in 171.43Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 198.778Âµs
>[kubeexec] ERROR 2018/06/08 08:47:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume stop vol_5db55c483f6984d9917cfe4b3c8b3cbc force] on glusterfs-storage-pg4xc: Err[command terminated with exit code 1]: Stdout [Error : Request timed out
>]: Stderr []
>[cmdexec] ERROR 2018/06/08 08:47:51 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:144: Unable to stop volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: 
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 125.079Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 239.169Âµs
>[kubeexec] DEBUG 2018/06/08 08:47:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:42 UTC; 2h 24min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4900c968_6ae4_11e8_bc4b_005056a5f18a.slice/docker-438fb5fb16a82a3e701b28fa1f1ef8623dc37ca06374c604e5781a2b2ab2e06d.scope/system.slice/glusterd.service
>           ââ  433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1106 /usr/sbin/glusterfsd -s 10.70.46.122 --volfile-id heketidbstorage.10.70.46.122.var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.122-var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.pid -S /var/run/gluster/9fa3ac3b12826912287adc88c26ac134.socket --brick-name /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_29e5b36d921cca1b0699871d9727c932/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.log --xlator-option *-posix.glusterd-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ14514 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f8994cdb09351d83e70da08261d99c64.socket --xlator-option *replicate*.node-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0
>
>Jun 08 06:23:40 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:42 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 08:47:51 Periodic health check status: node 70423cc0bcd044fe5ba8bbbd256a3e49 up=true
>[cmdexec] INFO 2018/06/08 08:47:51 Check Glusterd service status in node dhcp47-76.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 08:47:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script snapshot list vol_966a92ed3e4374a5e634f7b133c49e52 --xml
>Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
><cliOutput>
>  <opRet>0</opRet>
>  <opErrno>0</opErrno>
>  <opErrstr/>
>  <snapList>
>    <count>0</count>
>  </snapList>
></cliOutput>
>[kubeexec] ERROR 2018/06/08 08:47:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume delete vol_5db55c483f6984d9917cfe4b3c8b3cbc] on glusterfs-storage-pg4xc: Err[command terminated with exit code 1]: Stdout []: Stderr [volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Another transaction is in progress for vol_5db55c483f6984d9917cfe4b3c8b3cbc. Please try again after sometime.
>]
>[cmdexec] ERROR 2018/06/08 08:47:51 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:153: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Another transaction is in progress for vol_5db55c483f6984d9917cfe4b3c8b3cbc. Please try again after sometime.
>[heketi] ERROR 2018/06/08 08:47:51 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:525: Unable to delete volume: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Another transaction is in progress for vol_5db55c483f6984d9917cfe4b3c8b3cbc. Please try again after sometime.
>[heketi] ERROR 2018/06/08 08:47:51 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:429: Error executing delete volume: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Another transaction is in progress for vol_5db55c483f6984d9917cfe4b3c8b3cbc. Please try again after sometime.
>[asynchttp] INFO 2018/06/08 08:47:51 asynchttp.go:292: Completed job c5fa6ecf4c03887ea05cf8a2e6abfbd8 in 4m1.243706021s
>[heketi] ERROR 2018/06/08 08:47:51 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:1185: Delete Volume Failed: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Another transaction is in progress for vol_5db55c483f6984d9917cfe4b3c8b3cbc. Please try again after sometime.
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 5.907398ms
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 3.63494ms
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 689.986Âµs
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 1.167219ms
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 1.621921ms
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 586.362Âµs
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 608.912Âµs
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 635.131Âµs
>[negroni] Started GET /queue/c5fa6ecf4c03887ea05cf8a2e6abfbd8
>[negroni] Completed 500 Internal Server Error in 138.493Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 772.676Âµs
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 952.965Âµs
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 806.237Âµs
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 592.561Âµs
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 588.5Âµs
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 542.187Âµs
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 600.555Âµs
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 586.581Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 124.718Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 115.942Âµs
>[negroni] Completed 200 OK in 248.102Âµs
>[negroni] Started POST /volumes
>[negroni] Started POST /volumes
>[heketi] INFO 2018/06/08 08:47:52 Allocating brick set #0
>[negroni] Started POST /volumes
>[negroni] Started POST /volumes
>[negroni] Started POST /volumes
>[negroni] Started POST /volumes
>[negroni] Started POST /volumes
>[negroni] Started POST /volumes
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 139.399Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 174.425Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 73.924Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 150.491Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 155.116Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 183.273Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 193.428Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 90.728Âµs
>[negroni] Completed 200 OK in 240.724Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 167.275Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 106.245Âµs
>[negroni] Completed 200 OK in 79.542Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 211.138Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 134.186Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 125.629Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 202.691Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 144.816Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 76.681Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 136.784Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 155.835Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 110.733Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 237.688Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 141.193Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 204.923Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 128.552Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 145.129Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 67.016Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 125.02Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 134.727Âµs
>[negroni] Completed 200 OK in 98.239Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 138.7Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 143.029Âµs
>[negroni] Completed 200 OK in 201.191Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 143.828Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 196.31Âµs
>[negroni] Completed 200 OK in 200.471Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 212.384Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 242.855Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 105.711Âµs
>[negroni] Started DELETE /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 199.769Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 230.398Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 115.12Âµs
>[heketi] ERROR 2018/06/08 08:48:06 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:510: Unable to determine snapshot information from volume vol_9c85de66c12db0a72b4d16fe888ff74d: EOF
>[heketi] ERROR 2018/06/08 08:48:06 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:429: Error executing delete volume: Unable to determine snapshot information from volume vol_9c85de66c12db0a72b4d16fe888ff74d: EOF
>[kubeexec] DEBUG 2018/06/08 08:48:06 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script snapshot list vol_9c85de66c12db0a72b4d16fe888ff74d --xml
>Result: Error : Request timed out
>[kubeexec] DEBUG 2018/06/08 08:48:06 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 07:25:20 UTC; 1h 22min ago
>  Process: 828 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 830 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5cb2299_6aec_11e8_bc4b_005056a5f18a.slice/docker-7f9223abc9a8641cea68f374f4039235515978ee2dab970ac6cbef182c535426.scope/system.slice/glusterd.service
>           ââ  830 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1197 /usr/sbin/glusterfsd -s 10.70.47.76 --volfile-id heketidbstorage.10.70.47.76.var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.76-var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.pid -S /var/run/gluster/b6976f9ddc3abc90e1dfa91c35a7c611.socket --brick-name /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_54c06cdf52c6fe1e3a3f8a9d39cde33d/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.log --xlator-option *-posix.glusterd-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ12733 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/ec468453b11caed5259f5e1abcc65b44.socket --xlator-option *replicate*.node-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94
>           ââ12735 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/ec468453b11caed5259f5e1abcc65b44.socket --xlator-option *replicate*.node-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94
>
>Jun 08 07:25:17 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 07:25:20 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 08:48:06 Periodic health check status: node d942fe6c0ee5691b7cc263968f97b650 up=true
>[heketi] INFO 2018/06/08 08:48:06 Cleaned 0 nodes from health cache
>[heketi] INFO 2018/06/08 08:48:06 Starting Node Health Status refresh
>[cmdexec] INFO 2018/06/08 08:48:06 Check Glusterd service status in node dhcp46-187.lab.eng.blr.redhat.com
>[negroni] Completed 202 Accepted in 13.71483546s
>[asynchttp] INFO 2018/06/08 08:48:06 asynchttp.go:288: Started job a8d46afb5176808836420be86f6fdcd0
>[heketi] INFO 2018/06/08 08:48:06 Started async operation: Create Volume
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 155.827Âµs
>[heketi] INFO 2018/06/08 08:48:06 Creating brick d9145089dd3be60b9df2a82315900670
>[heketi] INFO 2018/06/08 08:48:06 Creating brick e416f1b7fd62ee9320cfd9d57705d34c
>[heketi] INFO 2018/06/08 08:48:06 Creating brick 14353da20ffb37550480dff24b915066
>[heketi] INFO 2018/06/08 08:48:06 Allocating brick set #0
>[kubeexec] DEBUG 2018/06/08 08:48:06 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir -p /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_e416f1b7fd62ee9320cfd9d57705d34c
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:48:06 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_96f1667f2f1ced2c5ef94772922be93b/tp_e416f1b7fd62ee9320cfd9d57705d34c --virtualsize 2097152K --name brick_e416f1b7fd62ee9320cfd9d57705d34c
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_e416f1b7fd62ee9320cfd9d57705d34c" created.
>[kubeexec] DEBUG 2018/06/08 08:48:06 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_e416f1b7fd62ee9320cfd9d57705d34c
>Result: meta-data=/dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_e416f1b7fd62ee9320cfd9d57705d34c isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 99.794Âµs
>[kubeexec] DEBUG 2018/06/08 08:48:07 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: awk "BEGIN {print \"/dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_e416f1b7fd62ee9320cfd9d57705d34c /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_e416f1b7fd62ee9320cfd9d57705d34c xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 139.788Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 104.526Âµs
>[kubeexec] DEBUG 2018/06/08 08:48:07 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_e416f1b7fd62ee9320cfd9d57705d34c /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_e416f1b7fd62ee9320cfd9d57705d34c
>Result: 
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 113.825Âµs
>[kubeexec] DEBUG 2018/06/08 08:48:07 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_e416f1b7fd62ee9320cfd9d57705d34c/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:48:07 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: chown :2000 /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_e416f1b7fd62ee9320cfd9d57705d34c/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:48:07 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: chmod 2775 /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_e416f1b7fd62ee9320cfd9d57705d34c/brick
>Result: 
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 116.471Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 107.47Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 75.671Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 170.129Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 153.152Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 133.491Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 86.945Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 163.249Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 230.442Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 119.724Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 78.723Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 140.105Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 123.1Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 121.244Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 73.304Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 224.765Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 202.291Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 285.351Âµs
>[negroni] Completed 200 OK in 101.248Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 148.383Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 164.212Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 129.935Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 84.304Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 157.554Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 140.018Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 162.886Âµs
>[negroni] Completed 200 OK in 201.331Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 168.516Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 190.381Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 276.958Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 146.745Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 174.982Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 203.658Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 235.261Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 86.974Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 145.345Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 215.355Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 251.348Âµs
>[negroni] Completed 200 OK in 143.246Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 194.833Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 213.279Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 150.069Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 192.265Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 182.138Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 190.951Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 329.187Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 213.095Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 194.148Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 136.87Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 238.852Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 186.472Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 131.188Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 140.015Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 191.641Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 140.475Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 133.677Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 182.715Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 184.402Âµs
>[negroni] Completed 200 OK in 128.589Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 142.246Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 117.186Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 219.701Âµs
>[negroni] Completed 200 OK in 110.646Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 305.718Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 227.362Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 180.705Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 141.975Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 276.478Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 220.639Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 150.818Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 202.341Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 203.985Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 150.449Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 146.717Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 187.435Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 230.998Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 186.471Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 169.326Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 142.543Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 277.484Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 219.224Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 145.434Âµs
>[negroni] Completed 200 OK in 140.199Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 186.003Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 247.994Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 196.925Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 134.782Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 129.818Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 186.698Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 181.089Âµs
>[negroni] Completed 200 OK in 99.629Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 178.956Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 208.466Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 176.865Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 189.045Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 199.442Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 122.312Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 185.576Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 145.866Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 211.052Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 216.285Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 211.402Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 95.032Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 129.076Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 176.002Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 183.703Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 178.998Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 180.063Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 229.662Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 208.629Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 132.336Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 185.001Âµs
>[negroni] Started POST /volumes
>[negroni] Started POST /volumes
>[negroni] Started POST /volumes
>[negroni] Started POST /volumes
>[negroni] Started POST /volumes
>[negroni] Started POST /volumes
>[negroni] Started POST /volumes
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 231.111Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 137.829Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 113.531Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 192.157Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 142.417Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 164.457Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 105.697Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 209.922Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 130.834Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 104.444Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 168.552Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 214.769Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 213.904Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 189.788Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 218.279Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 197.556Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 185.376Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 152.717Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 195.582Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 166.778Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 149.001Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 290.465Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 155.609Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 153.113Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 122.277Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 223.985Âµs
>[negroni] Completed 200 OK in 94.126Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 127.142Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 122.283Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 217.576Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 114.496Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 150.663Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 167.46Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 192.489Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 146.026Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 215.366Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 197.955Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 184.249Âµs
>[negroni] Completed 200 OK in 151.375Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 196.838Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 188.769Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 209.215Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 137.651Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 128.151Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 166.955Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 159.062Âµs
>[negroni] Completed 200 OK in 155.356Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 116.718Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 201.542Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 283.014Âµs
>[negroni] Completed 200 OK in 190.535Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 261.268Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 174.943Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 112.061Âµs
>[negroni] Completed 200 OK in 254.146Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 181.698Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 141.926Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 157.992Âµs
>[negroni] Completed 200 OK in 231.774Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 204.455Âµs
>[negroni] Started DELETE /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 190.411Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 227.372Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 184.499Âµs
>[kubeexec] ERROR 2018/06/08 08:48:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume stop vol_a4a6e4892da299f6c5634b8a2def697e force] on glusterfs-storage-vsh2m: Err[command terminated with exit code 1]: Stdout [Error : Request timed out
>]: Stderr []
>[cmdexec] ERROR 2018/06/08 08:48:51 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:144: Unable to stop volume vol_a4a6e4892da299f6c5634b8a2def697e: Unable to execute command on glusterfs-storage-vsh2m: 
>[kubeexec] DEBUG 2018/06/08 08:48:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:33 UTC; 2h 25min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48fed567_6ae4_11e8_bc4b_005056a5f18a.slice/docker-a506d9aaca4b440d184df40d01e90b2040811c8090bf4c19d892081307b28c0c.scope/system.slice/glusterd.service
>           ââ  433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1228 /usr/sbin/glusterfsd -s 10.70.46.187 --volfile-id heketidbstorage.10.70.46.187.var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.187-var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.pid -S /var/run/gluster/c08898c2e3b0190c62df3a9d4a013c92.socket --brick-name /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.log --xlator-option *-posix.glusterd-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ14272 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/8fb729a0bbd51b204491c603c84acaa4.socket --xlator-option *replicate*.node-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034
>
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Unit glusterd.service entered failed state.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: glusterd.service failed.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:33 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 08:48:51 Periodic health check status: node 278bd6b4e16a8e62ef15aaae22e6abc1 up=true
>[cmdexec] INFO 2018/06/08 08:48:51 Check Glusterd service status in node dhcp46-122.lab.eng.blr.redhat.com
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 96.618Âµs
>[negroni] Completed 202 Accepted in 58.843610259s
>[asynchttp] INFO 2018/06/08 08:48:51 asynchttp.go:288: Started job 6158d16fba595a299f350a673e859df3
>[heketi] INFO 2018/06/08 08:48:51 Started async operation: Create Volume
>[heketi] INFO 2018/06/08 08:48:51 Creating brick 30ba742d27672a5289fe7ff6bd5ef3ce
>[heketi] INFO 2018/06/08 08:48:51 Creating brick 2aca658dfb3ac9ba0fbf538dad4caa3b
>[heketi] INFO 2018/06/08 08:48:51 Creating brick c805e58953c8aa4c8e4d7298563713f7
>[heketi] INFO 2018/06/08 08:48:51 Allocating brick set #0
>[kubeexec] DEBUG 2018/06/08 08:48:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir -p /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_d9145089dd3be60b9df2a82315900670
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:48:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir -p /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_30ba742d27672a5289fe7ff6bd5ef3ce
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:48:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_b861af9cc67f5bf8d5a8f9e38f56b14e/tp_d9145089dd3be60b9df2a82315900670 --virtualsize 2097152K --name brick_d9145089dd3be60b9df2a82315900670
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_d9145089dd3be60b9df2a82315900670" created.
>[kubeexec] DEBUG 2018/06/08 08:48:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_9394bc70699b006c5460c9f654cf345f/tp_30ba742d27672a5289fe7ff6bd5ef3ce --virtualsize 2097152K --name brick_30ba742d27672a5289fe7ff6bd5ef3ce
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_30ba742d27672a5289fe7ff6bd5ef3ce" created.
>[kubeexec] DEBUG 2018/06/08 08:48:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_d9145089dd3be60b9df2a82315900670
>Result: meta-data=/dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_d9145089dd3be60b9df2a82315900670 isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:48:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_30ba742d27672a5289fe7ff6bd5ef3ce
>Result: meta-data=/dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_30ba742d27672a5289fe7ff6bd5ef3ce isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 189.982Âµs
>[kubeexec] DEBUG 2018/06/08 08:48:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: awk "BEGIN {print \"/dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_d9145089dd3be60b9df2a82315900670 /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_d9145089dd3be60b9df2a82315900670 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:48:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: awk "BEGIN {print \"/dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_30ba742d27672a5289fe7ff6bd5ef3ce /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_30ba742d27672a5289fe7ff6bd5ef3ce xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 95.984Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 102.291Âµs
>[kubeexec] DEBUG 2018/06/08 08:48:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_d9145089dd3be60b9df2a82315900670 /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_d9145089dd3be60b9df2a82315900670
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:48:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_30ba742d27672a5289fe7ff6bd5ef3ce /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_30ba742d27672a5289fe7ff6bd5ef3ce
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:48:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_d9145089dd3be60b9df2a82315900670/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:48:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_30ba742d27672a5289fe7ff6bd5ef3ce/brick
>Result: 
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 101.889Âµs
>[kubeexec] DEBUG 2018/06/08 08:48:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: chown :2000 /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_d9145089dd3be60b9df2a82315900670/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:48:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: chown :2001 /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_30ba742d27672a5289fe7ff6bd5ef3ce/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:48:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: chmod 2775 /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_d9145089dd3be60b9df2a82315900670/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:48:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: chmod 2775 /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_30ba742d27672a5289fe7ff6bd5ef3ce/brick
>Result: 
>[kubeexec] ERROR 2018/06/08 08:48:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume delete vol_a4a6e4892da299f6c5634b8a2def697e] on glusterfs-storage-vsh2m: Err[command terminated with exit code 1]: Stdout []: Stderr [volume delete: vol_a4a6e4892da299f6c5634b8a2def697e: failed: Another transaction is in progress for vol_a4a6e4892da299f6c5634b8a2def697e. Please try again after sometime.
>]
>[cmdexec] ERROR 2018/06/08 08:48:52 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:153: Unable to delete volume vol_a4a6e4892da299f6c5634b8a2def697e: Unable to execute command on glusterfs-storage-vsh2m: volume delete: vol_a4a6e4892da299f6c5634b8a2def697e: failed: Another transaction is in progress for vol_a4a6e4892da299f6c5634b8a2def697e. Please try again after sometime.
>[heketi] ERROR 2018/06/08 08:48:52 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:525: Unable to delete volume: Unable to delete volume vol_a4a6e4892da299f6c5634b8a2def697e: Unable to execute command on glusterfs-storage-vsh2m: volume delete: vol_a4a6e4892da299f6c5634b8a2def697e: failed: Another transaction is in progress for vol_a4a6e4892da299f6c5634b8a2def697e. Please try again after sometime.
>[heketi] ERROR 2018/06/08 08:48:52 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:429: Error executing delete volume: Unable to delete volume vol_a4a6e4892da299f6c5634b8a2def697e: Unable to execute command on glusterfs-storage-vsh2m: volume delete: vol_a4a6e4892da299f6c5634b8a2def697e: failed: Another transaction is in progress for vol_a4a6e4892da299f6c5634b8a2def697e. Please try again after sometime.
>[kubeexec] DEBUG 2018/06/08 08:48:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir -p /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_c805e58953c8aa4c8e4d7298563713f7
>Result: 
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 161.873Âµs
>[kubeexec] DEBUG 2018/06/08 08:48:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_b861af9cc67f5bf8d5a8f9e38f56b14e/tp_c805e58953c8aa4c8e4d7298563713f7 --virtualsize 2097152K --name brick_c805e58953c8aa4c8e4d7298563713f7
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_c805e58953c8aa4c8e4d7298563713f7" created.
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 169.447Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 81.457Âµs
>[kubeexec] DEBUG 2018/06/08 08:48:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_c805e58953c8aa4c8e4d7298563713f7
>Result: meta-data=/dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_c805e58953c8aa4c8e4d7298563713f7 isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:48:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: awk "BEGIN {print \"/dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_c805e58953c8aa4c8e4d7298563713f7 /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_c805e58953c8aa4c8e4d7298563713f7 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 144.728Âµs
>[kubeexec] DEBUG 2018/06/08 08:48:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_c805e58953c8aa4c8e4d7298563713f7 /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_c805e58953c8aa4c8e4d7298563713f7
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:48:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_c805e58953c8aa4c8e4d7298563713f7/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:48:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: chown :2001 /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_c805e58953c8aa4c8e4d7298563713f7/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:48:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: chmod 2775 /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_c805e58953c8aa4c8e4d7298563713f7/brick
>Result: 
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 131.853Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 193.385Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 103.91Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 118.457Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 104.489Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 105.37Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 96.408Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 189.506Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 116.587Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 218.772Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 124.826Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 170.532Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 159.531Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 254.985Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 133.502Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 138.568Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 113.613Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 164.395Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 72.901Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 208.292Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 179.579Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 182.445Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 123.946Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 253.852Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 190.083Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 170.309Âµs
>[negroni] Completed 200 OK in 102.908Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 207.091Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 142.213Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 145.194Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 102.374Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 153.877Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 194.657Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 120.354Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 265.949Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 120.418Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 123.112Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 166.793Âµs
>[negroni] Completed 200 OK in 184.823Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 117.271Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 146.42Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 183.638Âµs
>[negroni] Completed 200 OK in 167.148Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 159.823Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 174.322Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 208.134Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 149.833Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 126.26Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 166.216Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 170.67Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 153.142Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 257.035Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 131.611Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 132.596Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 134.169Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 183.166Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 134.687Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 183.636Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 183.629Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 256.724Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 191.999Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 212.134Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 222.388Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 196.758Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 204.123Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 133.951Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 224.398Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 145.021Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 215.948Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 175.299Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 113.117Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 108.276Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 257.188Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 322.678Âµs
>[negroni] Completed 200 OK in 363.191Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 124.742Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 166.217Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 183.788Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 99.714Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 203.604Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 165.129Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 168.109Âµs
>[negroni] Completed 200 OK in 161.573Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 143.381Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 182.692Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 165.368Âµs
>[negroni] Completed 200 OK in 192.336Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 141.011Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 194.016Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 246.461Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 167.986Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 130.672Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 147.648Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 104.226Âµs
>[negroni] Completed 200 OK in 295.185Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 253.978Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 295.771Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 189.828Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 142.342Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 210.049Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 199.605Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 146.346Âµs
>[negroni] Completed 200 OK in 149.575Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 212.527Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 174.916Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 189.822Âµs
>[negroni] Completed 200 OK in 136.242Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 189.072Âµs
>[negroni] Started POST /volumes
>[negroni] Started POST /volumes
>[negroni] Started POST /volumes
>[negroni] Started POST /volumes
>[negroni] Started POST /volumes
>[negroni] Started POST /volumes
>[negroni] Started POST /volumes
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 238.538Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 149.151Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 135.719Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 173.931Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 137.277Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 129.422Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 94.799Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 167.427Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 166.01Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 167.073Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 141.637Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 242.145Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 213.423Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 177.472Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 127.062Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 135.838Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 226.515Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 215.077Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 143.681Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 274.938Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 180.417Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 138.17Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 83.308Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 195.649Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 243.954Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 310.38Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 225.131Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 204.004Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 266.431Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 282.885Âµs
>[negroni] Completed 200 OK in 145.299Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 209.278Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 174.412Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 175.553Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 165.529Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 171.509Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 180.195Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 237.731Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 144.422Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 331.714Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 201.641Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 131.878Âµs
>[negroni] Completed 200 OK in 316.548Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 126.978Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 231.181Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 299.368Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 145.485Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 199.601Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 297.446Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 272.771Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 206.085Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 184.518Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 156.317Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 218.495Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 174.109Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 281.215Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 253.908Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 159.96Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 97.784Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 214.612Âµs
>[negroni] Started DELETE /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 153.274Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 258.268Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 150.592Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 133.383Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 129.25Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 173.978Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 156.794Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 200.39Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 166.832Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 173.792Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 128.432Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 138.712Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 150.523Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 220.44Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 80.156Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 194.158Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 176.675Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 153.322Âµs
>[negroni] Completed 200 OK in 88.655Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 137.241Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 203.788Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 206.181Âµs
>[negroni] Completed 200 OK in 296.607Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 185.361Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 208.166Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 138.731Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 99.751Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 261.511Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 139.352Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 146.556Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 136.919Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 356.464Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 199.812Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 158.138Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 355.428Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 145.206Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 246.722Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 240.918Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 198.711Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 281.681Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 234.794Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 133.415Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 76.564Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 137.206Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 339.994Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 211.362Âµs
>[negroni] Completed 200 OK in 158.716Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 227.931Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 206.898Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 187.319Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 115.882Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 246.708Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 149.944Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 362.717Âµs
>[negroni] Completed 200 OK in 181.605Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 160.969Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 224.808Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 271.925Âµs
>[negroni] Completed 200 OK in 281.741Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 196.522Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 191.774Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 179.411Âµs
>[negroni] Completed 200 OK in 268.378Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 143.621Âµs
>[kubeexec] ERROR 2018/06/08 08:49:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume stop vol_966a92ed3e4374a5e634f7b133c49e52 force] on glusterfs-storage-pg4xc: Err[command terminated with exit code 1]: Stdout [Error : Request timed out
>]: Stderr []
>[cmdexec] ERROR 2018/06/08 08:49:51 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:144: Unable to stop volume vol_966a92ed3e4374a5e634f7b133c49e52: Unable to execute command on glusterfs-storage-pg4xc: 
>[kubeexec] DEBUG 2018/06/08 08:49:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir -p /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_14353da20ffb37550480dff24b915066
>Result: 
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 126.068Âµs
>[kubeexec] DEBUG 2018/06/08 08:49:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_3a4297677881963e3f80124971d50eea/tp_14353da20ffb37550480dff24b915066 --virtualsize 2097152K --name brick_14353da20ffb37550480dff24b915066
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_14353da20ffb37550480dff24b915066" created.
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 114.65Âµs
>[negroni] Completed 200 OK in 92.551Âµs
>[kubeexec] DEBUG 2018/06/08 08:49:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_14353da20ffb37550480dff24b915066
>Result: meta-data=/dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_14353da20ffb37550480dff24b915066 isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:49:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: awk "BEGIN {print \"/dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_14353da20ffb37550480dff24b915066 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_14353da20ffb37550480dff24b915066 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 107.588Âµs
>[kubeexec] DEBUG 2018/06/08 08:49:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_14353da20ffb37550480dff24b915066 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_14353da20ffb37550480dff24b915066
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:49:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_14353da20ffb37550480dff24b915066/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:49:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: chown :2000 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_14353da20ffb37550480dff24b915066/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:49:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: chmod 2775 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_14353da20ffb37550480dff24b915066/brick
>Result: 
>[cmdexec] INFO 2018/06/08 08:49:52 Creating volume vol_ad1e5849e9566f1bcaa09cfb9c0b96ef replica 3
>[kubeexec] DEBUG 2018/06/08 08:49:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:42 UTC; 2h 26min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4900c968_6ae4_11e8_bc4b_005056a5f18a.slice/docker-438fb5fb16a82a3e701b28fa1f1ef8623dc37ca06374c604e5781a2b2ab2e06d.scope/system.slice/glusterd.service
>           ââ  433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1106 /usr/sbin/glusterfsd -s 10.70.46.122 --volfile-id heketidbstorage.10.70.46.122.var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.122-var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.pid -S /var/run/gluster/9fa3ac3b12826912287adc88c26ac134.socket --brick-name /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_29e5b36d921cca1b0699871d9727c932/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.log --xlator-option *-posix.glusterd-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ14514 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f8994cdb09351d83e70da08261d99c64.socket --xlator-option *replicate*.node-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0
>
>Jun 08 06:23:40 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:42 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 08:49:53 Periodic health check status: node 70423cc0bcd044fe5ba8bbbd256a3e49 up=true
>[cmdexec] INFO 2018/06/08 08:49:53 Check Glusterd service status in node dhcp47-76.lab.eng.blr.redhat.com
>[negroni] Completed 202 Accepted in 2m0.453436098s
>[asynchttp] INFO 2018/06/08 08:49:53 asynchttp.go:288: Started job d6cbe9a27af546f3c96e74f32e86cd27
>[heketi] INFO 2018/06/08 08:49:53 Started async operation: Create Volume
>[heketi] INFO 2018/06/08 08:49:53 Creating brick 1bb49691eccf328025855a91ee8cbc66
>[heketi] INFO 2018/06/08 08:49:53 Creating brick cc2483d7b49bc029b5200a024cac7535
>[heketi] INFO 2018/06/08 08:49:53 Creating brick 2ed96343627983cf9667b7cee4052d17
>[heketi] INFO 2018/06/08 08:49:53 Allocating brick set #0
>[kubeexec] DEBUG 2018/06/08 08:49:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 07:25:20 UTC; 1h 24min ago
>  Process: 828 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 830 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5cb2299_6aec_11e8_bc4b_005056a5f18a.slice/docker-7f9223abc9a8641cea68f374f4039235515978ee2dab970ac6cbef182c535426.scope/system.slice/glusterd.service
>           ââ  830 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1197 /usr/sbin/glusterfsd -s 10.70.47.76 --volfile-id heketidbstorage.10.70.47.76.var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.76-var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.pid -S /var/run/gluster/b6976f9ddc3abc90e1dfa91c35a7c611.socket --brick-name /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_54c06cdf52c6fe1e3a3f8a9d39cde33d/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.log --xlator-option *-posix.glusterd-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ12733 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/ec468453b11caed5259f5e1abcc65b44.socket --xlator-option *replicate*.node-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94
>           ââ12735 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/ec468453b11caed5259f5e1abcc65b44.socket --xlator-option *replicate*.node-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94
>
>Jun 08 07:25:17 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 07:25:20 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 08:49:53 Periodic health check status: node d942fe6c0ee5691b7cc263968f97b650 up=true
>[heketi] INFO 2018/06/08 08:49:53 Cleaned 0 nodes from health cache
>[heketi] INFO 2018/06/08 08:49:53 Starting Node Health Status refresh
>[cmdexec] INFO 2018/06/08 08:49:53 Check Glusterd service status in node dhcp46-187.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 08:49:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir -p /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_2aca658dfb3ac9ba0fbf538dad4caa3b
>Result: 
>[negroni] Completed 202 Accepted in 2m0.552281252s
>[asynchttp] INFO 2018/06/08 08:49:53 asynchttp.go:288: Started job 078bf2d1330026527a20bcfa4ebc6028
>[heketi] INFO 2018/06/08 08:49:53 Started async operation: Create Volume
>[heketi] INFO 2018/06/08 08:49:53 Creating brick df5559fdf15f6372e19e4e9f8bc1f129
>[heketi] INFO 2018/06/08 08:49:53 Creating brick dad9ada287c446b0011af8dc964060e1
>[heketi] INFO 2018/06/08 08:49:53 Creating brick 3dbab6cd698d01ea7f00dbd81329643a
>[heketi] INFO 2018/06/08 08:49:53 Allocating brick set #0
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 144.863Âµs
>[kubeexec] DEBUG 2018/06/08 08:49:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir -p /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_2ed96343627983cf9667b7cee4052d17
>Result: 
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 105.476Âµs
>[negroni] Completed 200 OK in 83.354Âµs
>[kubeexec] DEBUG 2018/06/08 08:49:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_66a9af9f6bac95f9d8d556a2f14c29d3/tp_2aca658dfb3ac9ba0fbf538dad4caa3b --virtualsize 2097152K --name brick_2aca658dfb3ac9ba0fbf538dad4caa3b
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_2aca658dfb3ac9ba0fbf538dad4caa3b" created.
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 165.668Âµs
>[kubeexec] DEBUG 2018/06/08 08:49:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_2aca658dfb3ac9ba0fbf538dad4caa3b
>Result: meta-data=/dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_2aca658dfb3ac9ba0fbf538dad4caa3b isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:49:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_9394bc70699b006c5460c9f654cf345f/tp_2ed96343627983cf9667b7cee4052d17 --virtualsize 2097152K --name brick_2ed96343627983cf9667b7cee4052d17
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_2ed96343627983cf9667b7cee4052d17" created.
>[kubeexec] DEBUG 2018/06/08 08:49:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: awk "BEGIN {print \"/dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_2aca658dfb3ac9ba0fbf538dad4caa3b /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_2aca658dfb3ac9ba0fbf538dad4caa3b xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:49:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_2ed96343627983cf9667b7cee4052d17
>Result: meta-data=/dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_2ed96343627983cf9667b7cee4052d17 isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[negroni] Started POST /volumes
>[kubeexec] DEBUG 2018/06/08 08:49:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_2aca658dfb3ac9ba0fbf538dad4caa3b /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_2aca658dfb3ac9ba0fbf538dad4caa3b
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:49:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: awk "BEGIN {print \"/dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_2ed96343627983cf9667b7cee4052d17 /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_2ed96343627983cf9667b7cee4052d17 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:49:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_2aca658dfb3ac9ba0fbf538dad4caa3b/brick
>Result: 
>[negroni] Started POST /volumes
>[kubeexec] DEBUG 2018/06/08 08:49:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: chown :2001 /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_2aca658dfb3ac9ba0fbf538dad4caa3b/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:49:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_2ed96343627983cf9667b7cee4052d17 /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_2ed96343627983cf9667b7cee4052d17
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:49:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: chmod 2775 /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_2aca658dfb3ac9ba0fbf538dad4caa3b/brick
>Result: 
>[cmdexec] INFO 2018/06/08 08:49:54 Creating volume vol_15e0122e942fc41f80666a3714670682 replica 3
>[kubeexec] DEBUG 2018/06/08 08:49:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_2ed96343627983cf9667b7cee4052d17/brick
>Result: 
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 154.48Âµs
>[kubeexec] DEBUG 2018/06/08 08:49:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: chown :2002 /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_2ed96343627983cf9667b7cee4052d17/brick
>Result: 
>[negroni] Started POST /volumes
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 118.576Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 76.056Âµs
>[kubeexec] ERROR 2018/06/08 08:49:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume delete vol_966a92ed3e4374a5e634f7b133c49e52] on glusterfs-storage-pg4xc: Err[command terminated with exit code 1]: Stdout []: Stderr [volume delete: vol_966a92ed3e4374a5e634f7b133c49e52: failed: Another transaction is in progress for vol_966a92ed3e4374a5e634f7b133c49e52. Please try again after sometime.
>]
>[cmdexec] ERROR 2018/06/08 08:49:54 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:153: Unable to delete volume vol_966a92ed3e4374a5e634f7b133c49e52: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_966a92ed3e4374a5e634f7b133c49e52: failed: Another transaction is in progress for vol_966a92ed3e4374a5e634f7b133c49e52. Please try again after sometime.
>[heketi] ERROR 2018/06/08 08:49:54 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:525: Unable to delete volume: Unable to delete volume vol_966a92ed3e4374a5e634f7b133c49e52: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_966a92ed3e4374a5e634f7b133c49e52: failed: Another transaction is in progress for vol_966a92ed3e4374a5e634f7b133c49e52. Please try again after sometime.
>[heketi] ERROR 2018/06/08 08:49:54 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:429: Error executing delete volume: Unable to delete volume vol_966a92ed3e4374a5e634f7b133c49e52: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_966a92ed3e4374a5e634f7b133c49e52: failed: Another transaction is in progress for vol_966a92ed3e4374a5e634f7b133c49e52. Please try again after sometime.
>[kubeexec] DEBUG 2018/06/08 08:49:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: chmod 2775 /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_2ed96343627983cf9667b7cee4052d17/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:49:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir -p /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_cc2483d7b49bc029b5200a024cac7535
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:49:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir -p /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_df5559fdf15f6372e19e4e9f8bc1f129
>Result: 
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 111.932Âµs
>[negroni] Started POST /volumes
>[negroni] Started POST /volumes
>[kubeexec] DEBUG 2018/06/08 08:49:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_3a4297677881963e3f80124971d50eea/tp_cc2483d7b49bc029b5200a024cac7535 --virtualsize 2097152K --name brick_cc2483d7b49bc029b5200a024cac7535
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_cc2483d7b49bc029b5200a024cac7535" created.
>[kubeexec] DEBUG 2018/06/08 08:49:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_96f1667f2f1ced2c5ef94772922be93b/tp_df5559fdf15f6372e19e4e9f8bc1f129 --virtualsize 2097152K --name brick_df5559fdf15f6372e19e4e9f8bc1f129
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_df5559fdf15f6372e19e4e9f8bc1f129" created.
>[kubeexec] DEBUG 2018/06/08 08:49:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_cc2483d7b49bc029b5200a024cac7535
>Result: meta-data=/dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_cc2483d7b49bc029b5200a024cac7535 isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:49:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_df5559fdf15f6372e19e4e9f8bc1f129
>Result: meta-data=/dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_df5559fdf15f6372e19e4e9f8bc1f129 isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:49:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: awk "BEGIN {print \"/dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_cc2483d7b49bc029b5200a024cac7535 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_cc2483d7b49bc029b5200a024cac7535 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[negroni] Started POST /volumes
>[kubeexec] DEBUG 2018/06/08 08:49:55 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: awk "BEGIN {print \"/dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_df5559fdf15f6372e19e4e9f8bc1f129 /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_df5559fdf15f6372e19e4e9f8bc1f129 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:49:55 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_cc2483d7b49bc029b5200a024cac7535 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_cc2483d7b49bc029b5200a024cac7535
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:49:55 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_df5559fdf15f6372e19e4e9f8bc1f129 /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_df5559fdf15f6372e19e4e9f8bc1f129
>Result: 
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 145.839Âµs
>[kubeexec] DEBUG 2018/06/08 08:49:55 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_cc2483d7b49bc029b5200a024cac7535/brick
>Result: 
>[negroni] Started POST /volumes
>[kubeexec] DEBUG 2018/06/08 08:49:55 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_df5559fdf15f6372e19e4e9f8bc1f129/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:49:55 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: chown :2002 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_cc2483d7b49bc029b5200a024cac7535/brick
>Result: 
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 124.92Âµs
>[negroni] Completed 200 OK in 120.617Âµs
>[kubeexec] DEBUG 2018/06/08 08:49:55 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: chown :2003 /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_df5559fdf15f6372e19e4e9f8bc1f129/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:49:55 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: chmod 2775 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_cc2483d7b49bc029b5200a024cac7535/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:49:55 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: chmod 2775 /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_df5559fdf15f6372e19e4e9f8bc1f129/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:49:55 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir -p /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_3dbab6cd698d01ea7f00dbd81329643a
>Result: 
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 118.397Âµs
>[kubeexec] DEBUG 2018/06/08 08:49:55 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_66a9af9f6bac95f9d8d556a2f14c29d3/tp_3dbab6cd698d01ea7f00dbd81329643a --virtualsize 2097152K --name brick_3dbab6cd698d01ea7f00dbd81329643a
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_3dbab6cd698d01ea7f00dbd81329643a" created.
>[kubeexec] DEBUG 2018/06/08 08:49:55 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_3dbab6cd698d01ea7f00dbd81329643a
>Result: meta-data=/dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_3dbab6cd698d01ea7f00dbd81329643a isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:49:55 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: awk "BEGIN {print \"/dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_3dbab6cd698d01ea7f00dbd81329643a /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_3dbab6cd698d01ea7f00dbd81329643a xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 1.299228ms
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 1.386327ms
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 880.047Âµs
>[negroni] Started GET /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 200 OK in 627.511Âµs
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 932.96Âµs
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 926.287Âµs
>[negroni] Started GET /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 200 OK in 572.183Âµs
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 543.119Âµs
>[kubeexec] DEBUG 2018/06/08 08:49:56 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_3dbab6cd698d01ea7f00dbd81329643a /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_3dbab6cd698d01ea7f00dbd81329643a
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:49:56 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_3dbab6cd698d01ea7f00dbd81329643a/brick
>Result: 
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 102.499Âµs
>[kubeexec] DEBUG 2018/06/08 08:49:56 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: chown :2003 /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_3dbab6cd698d01ea7f00dbd81329643a/brick
>Result: 
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 197.851Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 114.18Âµs
>[kubeexec] DEBUG 2018/06/08 08:49:56 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: chmod 2775 /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_3dbab6cd698d01ea7f00dbd81329643a/brick
>Result: 
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 108.489Âµs
>[negroni] Started POST /volumes
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 127.224Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 136.614Âµs
>[negroni] Completed 200 OK in 97.101Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 130.635Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 131.461Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 165.068Âµs
>[negroni] Completed 200 OK in 115.288Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 129.844Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 114.063Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 285.631Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 119.808Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 132.393Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 124.375Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 121.503Âµs
>[negroni] Completed 200 OK in 101.166Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 137.964Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 140.875Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 136.889Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 200.337Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 113.678Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 219.092Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 243.852Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 216.688Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 145.562Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 159.849Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 150.642Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 239.655Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 269.511Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 161.715Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 266.712Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 122.747Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 145.09Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 201.278Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 160.794Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 122.131Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 204.978Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 193.649Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 196.515Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 130.693Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 139.646Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 144.771Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 207.109Âµs
>[negroni] Completed 200 OK in 89.202Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 157.795Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 192.828Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 199.608Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 107.47Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 204.682Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 213.231Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 205.997Âµs
>[negroni] Completed 200 OK in 366.908Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 175.009Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 162.795Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 207.541Âµs
>[negroni] Completed 200 OK in 103.037Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 156.824Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 172.545Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 242.725Âµs
>[negroni] Completed 200 OK in 99.733Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 133.764Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 139.866Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 188.043Âµs
>[negroni] Completed 200 OK in 147.179Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 168.722Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 219.098Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 189.528Âµs
>[negroni] Completed 200 OK in 158.568Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 207.725Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 183.473Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 145.644Âµs
>[negroni] Completed 200 OK in 142.742Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 140.016Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 155.777Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 223.982Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 152.345Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 268.304Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 199.822Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 194.215Âµs
>[negroni] Completed 200 OK in 139.445Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 185.962Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 180.673Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 213.216Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 200.942Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 204.035Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 188.762Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 217.242Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 199.072Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 182.074Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 180.484Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 205.992Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 200.918Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 206.925Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 220.038Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 333.088Âµs
>[negroni] Completed 200 OK in 163.938Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 262.468Âµs
>[negroni] Started DELETE /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 247.578Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 151.065Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 89.95Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 131.309Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 203.819Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 206.978Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 151.19Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 214.222Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 229.851Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 195.909Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 130.445Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 181.809Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 343.511Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 247.528Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 166.062Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 181.243Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 158.815Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 276.551Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 134.648Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 186.638Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 182.509Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 232.132Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 160.949Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 381.713Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 201.159Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 308.114Âµs
>[negroni] Completed 200 OK in 136.122Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 170.725Âµs
>[negroni] Started POST /volumes
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 144.406Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 135.314Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 93.425Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 131.074Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 204.479Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 244.048Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 138.616Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 306.917Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 175.82Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 157.472Âµs
>[negroni] Completed 200 OK in 260.586Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 276.451Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 213.297Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 131.937Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 87.492Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 130.517Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 188.348Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 172.976Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 107.246Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 217.452Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 166.432Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 341.581Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 143.098Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 260.105Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 250.244Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 185.411Âµs
>[negroni] Completed 200 OK in 151.782Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 168.933Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 287.595Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 221.089Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 210.389Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 254.764Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 202.825Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 155.516Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 95.382Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 134.072Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 178.343Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 194.252Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 157.692Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 187.812Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 237.135Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 243.734Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 169.242Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 199.022Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 223.935Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 144.096Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 172.799Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 230.052Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 152.841Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 174.653Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 114.448Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 133.879Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 157.864Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 220.651Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 241.695Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 223.716Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 232.338Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 189.838Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 111.086Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 258.511Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 208.302Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 157.228Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 152.009Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 197.356Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 212.829Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 228.661Âµs
>[negroni] Completed 200 OK in 165.402Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 182.816Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 191.809Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 191.796Âµs
>[negroni] Completed 200 OK in 139.926Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 159.168Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 236.268Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 183.789Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 166.546Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 204.672Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 140.15Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 254.146Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 165.759Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 281.731Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 182.569Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 183.073Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 256.612Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 223.454Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 330.111Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 140.542Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 172.319Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 201.852Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 148.018Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 190.744Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 169.183Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 162.761Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 192.896Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 223.325Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 137.525Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 135.709Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 210.295Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 208.459Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 119.317Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 257.745Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 194.331Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 183.535Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 121.383Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 254.377Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 149.62Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 201.882Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 125.733Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 247.141Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 205.564Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 137.945Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 73.777Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 188.787Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 203.358Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 210.435Âµs
>[negroni] Completed 200 OK in 251.834Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 195.325Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 193.789Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 192.562Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 150.863Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 206.701Âµs
>[negroni] Started POST /volumes
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 194.066Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 179.959Âµs
>[negroni] Completed 200 OK in 203.978Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 150.686Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 137.062Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 205.129Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 167.846Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 211.201Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 201.755Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 193.219Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 328.467Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 175.522Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 158.329Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 180.293Âµs
>[negroni] Completed 200 OK in 105.035Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 227.43Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 145.297Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 130.578Âµs
>[negroni] Completed 200 OK in 138.276Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 150.135Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 131.027Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 201.785Âµs
>[negroni] Completed 200 OK in 160.349Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 212.255Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 191.212Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 195.59Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 166.357Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 149.291Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 157.235Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 244.358Âµs
>[negroni] Completed 200 OK in 103.257Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 203.305Âµs
>[negroni] Started DELETE /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 206.669Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 186.409Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 134.907Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 217.168Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 192.656Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 194.821Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 195.142Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 186.871Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 191.135Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 183.361Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 147.045Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 234.215Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 202.171Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 196.353Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 88.133Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 242.838Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 143.4Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 150.844Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 117.557Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 150.507Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 256.451Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 233.978Âµs
>[negroni] Completed 200 OK in 269.038Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 166.562Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 200.218Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 255.888Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 569.996Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 134.955Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 200.261Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 191.955Âµs
>[negroni] Completed 200 OK in 82.777Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 187.498Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 147.717Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 253.008Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 154.859Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 128.249Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 201.081Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 225.758Âµs
>[negroni] Completed 200 OK in 161.992Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 343.743Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 218.071Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 148.789Âµs
>[negroni] Completed 200 OK in 219.403Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 135.655Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 257.718Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 181.129Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 202.199Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 203.676Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 252.445Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 207.861Âµs
>[negroni] Completed 200 OK in 160.335Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 167.361Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 204.023Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 193.788Âµs
>[negroni] Completed 200 OK in 146.175Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 214.432Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 154.35Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 191.428Âµs
>[negroni] Completed 200 OK in 142.679Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 291.214Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 143.498Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 274.018Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 125.853Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 271.948Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 227.951Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 131.511Âµs
>[negroni] Completed 200 OK in 106.476Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 215.195Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 193.962Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 200.068Âµs
>[negroni] Completed 200 OK in 164.683Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 211.755Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 179.327Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 165.592Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 172.171Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 152.413Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 152.95Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 212.486Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 156.353Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 149.562Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 208.645Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 206.775Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 202.525Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 242.285Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 233.648Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 281.245Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 134.859Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 219.688Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 123.726Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 141.944Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 122.913Âµs
>[negroni] Started POST /volumes
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 129.992Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 191.362Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 122.255Âµs
>[negroni] Completed 200 OK in 1.053625ms
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 207.148Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 224.021Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 208.858Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 135.126Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 189.896Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 223.188Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 129.731Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 178.035Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 178.819Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 196.712Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 160.436Âµs
>[negroni] Completed 200 OK in 384.517Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 211.481Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 193.521Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 197.576Âµs
>[negroni] Completed 200 OK in 296.628Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 288.157Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 192.675Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 175.284Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 145.663Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 181.499Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 296.231Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 211.488Âµs
>[negroni] Completed 200 OK in 139.386Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 192.398Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 230.972Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 190.235Âµs
>[negroni] Completed 200 OK in 249.171Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 205.089Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 236.455Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 226.648Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 440.58Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 177.677Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 226.445Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 149.084Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 144.922Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 195.549Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 225.155Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 188.795Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 104.616Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 128.123Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 239.132Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 125.034Âµs
>[negroni] Completed 200 OK in 80.839Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 133.583Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 168.478Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 177.043Âµs
>[negroni] Completed 200 OK in 183.936Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 243.805Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 224.982Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 130.013Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 298.719Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 185.776Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 197.988Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 185.555Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 102.162Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 188.578Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 183.718Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 220.715Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 124.946Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 232.618Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 193.202Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 220.414Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 234.909Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 262.624Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 253.592Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 158.039Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 263.858Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 140.119Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 129.887Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 249.181Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 189.848Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 213.655Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 203.405Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 265.978Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 215.008Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 175.691Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 207.948Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 145.147Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 136.282Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 253.711Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 145.143Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 214.699Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 289.501Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 294.584Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 328.203Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 243.911Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 124.468Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 177.561Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 149.528Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 199.155Âµs
>[negroni] Completed 200 OK in 259.224Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 234.371Âµs
>[kubeexec] ERROR 2018/06/08 08:51:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume create vol_ad1e5849e9566f1bcaa09cfb9c0b96ef replica 3 10.70.46.187:/var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_d9145089dd3be60b9df2a82315900670/brick 10.70.46.122:/var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_14353da20ffb37550480dff24b915066/brick 10.70.47.76:/var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_e416f1b7fd62ee9320cfd9d57705d34c/brick] on glusterfs-storage-vsh2m: Err[command terminated with exit code 1]: Stdout [Error : Request timed out
>]: Stderr []
>[kubeexec] DEBUG 2018/06/08 08:51:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir -p /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_1bb49691eccf328025855a91ee8cbc66
>Result: 
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 188.059Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 225.199Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 103.759Âµs
>[kubeexec] DEBUG 2018/06/08 08:51:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_b861af9cc67f5bf8d5a8f9e38f56b14e/tp_1bb49691eccf328025855a91ee8cbc66 --virtualsize 2097152K --name brick_1bb49691eccf328025855a91ee8cbc66
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_1bb49691eccf328025855a91ee8cbc66" created.
>[kubeexec] DEBUG 2018/06/08 08:51:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_1bb49691eccf328025855a91ee8cbc66
>Result: meta-data=/dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_1bb49691eccf328025855a91ee8cbc66 isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 162.556Âµs
>[kubeexec] DEBUG 2018/06/08 08:51:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: awk "BEGIN {print \"/dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_1bb49691eccf328025855a91ee8cbc66 /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_1bb49691eccf328025855a91ee8cbc66 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:51:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_1bb49691eccf328025855a91ee8cbc66 /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_1bb49691eccf328025855a91ee8cbc66
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:51:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_1bb49691eccf328025855a91ee8cbc66/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:51:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: chown :2002 /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_1bb49691eccf328025855a91ee8cbc66/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:51:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: chmod 2775 /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_1bb49691eccf328025855a91ee8cbc66/brick
>Result: 
>[cmdexec] INFO 2018/06/08 08:51:54 Creating volume vol_9f9fb17746d0aad637b132875d2744e5 replica 3
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 146.5Âµs
>[kubeexec] DEBUG 2018/06/08 08:51:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:33 UTC; 2h 28min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48fed567_6ae4_11e8_bc4b_005056a5f18a.slice/docker-a506d9aaca4b440d184df40d01e90b2040811c8090bf4c19d892081307b28c0c.scope/system.slice/glusterd.service
>           ââ  433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1228 /usr/sbin/glusterfsd -s 10.70.46.187 --volfile-id heketidbstorage.10.70.46.187.var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.187-var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.pid -S /var/run/gluster/c08898c2e3b0190c62df3a9d4a013c92.socket --brick-name /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.log --xlator-option *-posix.glusterd-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ14272 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/8fb729a0bbd51b204491c603c84acaa4.socket --xlator-option *replicate*.node-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034
>
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Unit glusterd.service entered failed state.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: glusterd.service failed.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:33 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 08:51:54 Periodic health check status: node 278bd6b4e16a8e62ef15aaae22e6abc1 up=true
>[cmdexec] INFO 2018/06/08 08:51:54 Check Glusterd service status in node dhcp46-122.lab.eng.blr.redhat.com
>[negroni] Completed 202 Accepted in 4m1.895668064s
>[asynchttp] INFO 2018/06/08 08:51:54 asynchttp.go:288: Started job d3134ebfac4ef9bd8dbf824bd7bf873a
>[heketi] INFO 2018/06/08 08:51:54 Started async operation: Create Volume
>[heketi] INFO 2018/06/08 08:51:54 Creating brick c6935061d1c9dbb05f316e3d87080a38
>[heketi] INFO 2018/06/08 08:51:54 Creating brick f4fd75475848eb3cf3c5f985495d64f3
>[heketi] INFO 2018/06/08 08:51:54 Creating brick 5a17b5c320a4f4efb036a3001a05544b
>[heketi] INFO 2018/06/08 08:51:54 Allocating brick set #0
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 114.506Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 300.812Âµs
>[kubeexec] DEBUG 2018/06/08 08:51:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir -p /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_dad9ada287c446b0011af8dc964060e1
>Result: 
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 189.42Âµs
>[kubeexec] DEBUG 2018/06/08 08:51:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_d389f0278a774bd7443a09af960961d8/tp_dad9ada287c446b0011af8dc964060e1 --virtualsize 2097152K --name brick_dad9ada287c446b0011af8dc964060e1
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_dad9ada287c446b0011af8dc964060e1" created.
>[kubeexec] DEBUG 2018/06/08 08:51:55 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_dad9ada287c446b0011af8dc964060e1
>Result: meta-data=/dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_dad9ada287c446b0011af8dc964060e1 isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:51:55 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: awk "BEGIN {print \"/dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_dad9ada287c446b0011af8dc964060e1 /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_dad9ada287c446b0011af8dc964060e1 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:51:55 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_dad9ada287c446b0011af8dc964060e1 /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_dad9ada287c446b0011af8dc964060e1
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:51:55 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_dad9ada287c446b0011af8dc964060e1/brick
>Result: 
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 150.178Âµs
>[kubeexec] DEBUG 2018/06/08 08:51:55 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: chown :2003 /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_dad9ada287c446b0011af8dc964060e1/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:51:55 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: chmod 2775 /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_dad9ada287c446b0011af8dc964060e1/brick
>Result: 
>[cmdexec] INFO 2018/06/08 08:51:55 Creating volume vol_b99532640a5201d243193159ee762ae4 replica 3
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 101.944Âµs
>[negroni] Completed 200 OK in 64.643Âµs
>[kubeexec] ERROR 2018/06/08 08:51:55 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume create vol_15e0122e942fc41f80666a3714670682 replica 3 10.70.47.76:/var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_30ba742d27672a5289fe7ff6bd5ef3ce/brick 10.70.46.187:/var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_c805e58953c8aa4c8e4d7298563713f7/brick 10.70.46.122:/var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_2aca658dfb3ac9ba0fbf538dad4caa3b/brick] on glusterfs-storage-gxp7c: Err[command terminated with exit code 1]: Stdout [Error : Request timed out
>]: Stderr []
>[kubeexec] ERROR 2018/06/08 08:51:55 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume stop vol_ad1e5849e9566f1bcaa09cfb9c0b96ef force] on glusterfs-storage-vsh2m: Err[command terminated with exit code 1]: Stdout []: Stderr [volume stop: vol_ad1e5849e9566f1bcaa09cfb9c0b96ef: failed: Another transaction is in progress for vol_ad1e5849e9566f1bcaa09cfb9c0b96ef. Please try again after sometime.
>]
>[cmdexec] ERROR 2018/06/08 08:51:55 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:144: Unable to stop volume vol_ad1e5849e9566f1bcaa09cfb9c0b96ef: Unable to execute command on glusterfs-storage-vsh2m: volume stop: vol_ad1e5849e9566f1bcaa09cfb9c0b96ef: failed: Another transaction is in progress for vol_ad1e5849e9566f1bcaa09cfb9c0b96ef. Please try again after sometime.
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 111.793Âµs
>[kubeexec] DEBUG 2018/06/08 08:51:55 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir -p /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_f4fd75475848eb3cf3c5f985495d64f3
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:51:56 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_b861af9cc67f5bf8d5a8f9e38f56b14e/tp_f4fd75475848eb3cf3c5f985495d64f3 --virtualsize 2097152K --name brick_f4fd75475848eb3cf3c5f985495d64f3
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_f4fd75475848eb3cf3c5f985495d64f3" created.
>[kubeexec] DEBUG 2018/06/08 08:51:56 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_f4fd75475848eb3cf3c5f985495d64f3
>Result: meta-data=/dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_f4fd75475848eb3cf3c5f985495d64f3 isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:51:56 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: awk "BEGIN {print \"/dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_f4fd75475848eb3cf3c5f985495d64f3 /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_f4fd75475848eb3cf3c5f985495d64f3 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:51:56 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_f4fd75475848eb3cf3c5f985495d64f3 /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_f4fd75475848eb3cf3c5f985495d64f3
>Result: 
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 117.725Âµs
>[kubeexec] DEBUG 2018/06/08 08:51:56 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_f4fd75475848eb3cf3c5f985495d64f3/brick
>Result: 
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 137.185Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 70.448Âµs
>[kubeexec] DEBUG 2018/06/08 08:51:56 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: chown :2004 /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_f4fd75475848eb3cf3c5f985495d64f3/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:51:56 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: chmod 2775 /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_f4fd75475848eb3cf3c5f985495d64f3/brick
>Result: 
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 112.727Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 149.542Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 136.177Âµs
>[negroni] Completed 200 OK in 98.126Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 109.317Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 181.523Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 130.809Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 99.73Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 180.331Âµs
>[negroni] Started POST /volumes
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 108.609Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 169.406Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 125.169Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 195.254Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 181.302Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 138.738Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 129.143Âµs
>[kubeexec] DEBUG 2018/06/08 08:52:00 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir -p /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_c6935061d1c9dbb05f316e3d87080a38
>Result: 
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 135.642Âµs
>[kubeexec] DEBUG 2018/06/08 08:52:01 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_96f1667f2f1ced2c5ef94772922be93b/tp_c6935061d1c9dbb05f316e3d87080a38 --virtualsize 2097152K --name brick_c6935061d1c9dbb05f316e3d87080a38
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_c6935061d1c9dbb05f316e3d87080a38" created.
>[kubeexec] DEBUG 2018/06/08 08:52:01 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_c6935061d1c9dbb05f316e3d87080a38
>Result: meta-data=/dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_c6935061d1c9dbb05f316e3d87080a38 isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:52:01 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: awk "BEGIN {print \"/dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_c6935061d1c9dbb05f316e3d87080a38 /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_c6935061d1c9dbb05f316e3d87080a38 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 111.224Âµs
>[kubeexec] DEBUG 2018/06/08 08:52:01 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_c6935061d1c9dbb05f316e3d87080a38 /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_c6935061d1c9dbb05f316e3d87080a38
>Result: 
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 120.692Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 90.544Âµs
>[kubeexec] DEBUG 2018/06/08 08:52:01 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_c6935061d1c9dbb05f316e3d87080a38/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:52:01 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: chown :2004 /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_c6935061d1c9dbb05f316e3d87080a38/brick
>Result: 
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 135.658Âµs
>[kubeexec] DEBUG 2018/06/08 08:52:01 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: chmod 2775 /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_c6935061d1c9dbb05f316e3d87080a38/brick
>Result: 
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 144.345Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 165.147Âµs
>[negroni] Completed 200 OK in 184.125Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 126.69Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 200.922Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 129.112Âµs
>[negroni] Completed 200 OK in 76.161Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 123.844Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 126.507Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 150.084Âµs
>[negroni] Completed 200 OK in 75.173Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 111.054Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 185.359Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 248.748Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 137.136Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 150.465Âµs
>[negroni] Started DELETE /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 195.592Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 229.303Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 97.168Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 192.973Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 215.616Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 241.182Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 122.733Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 192.461Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 173.807Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 184.634Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 157.943Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 154.27Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 266.731Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 260.124Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 133.762Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 204.081Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 271.971Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 195.412Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 137.229Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 231.044Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 198.078Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 184.035Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 106.5Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 168.019Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 186.639Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 199.322Âµs
>[negroni] Completed 200 OK in 173.768Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 243.591Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 148.229Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 193.809Âµs
>[negroni] Completed 200 OK in 260.725Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 142.809Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 221.228Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 162.908Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 132.759Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 195.238Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 191.758Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 219.221Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 153.935Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 188.355Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 214.799Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 181.029Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 309.211Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 263.408Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 205.175Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 193.279Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 137.896Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 206.835Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 136.468Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 137.415Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 211.015Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 195.044Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 310.435Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 248.805Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 167.072Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 211.908Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 191.342Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 148.178Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 226.029Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 164.897Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 197.166Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 167.435Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 130.942Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 130.206Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 151.172Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 183.324Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 152.642Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 153.291Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 199.018Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 227.028Âµs
>[negroni] Completed 200 OK in 141.699Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 233.475Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 263.651Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 202.898Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 142.366Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 185.509Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 198.289Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 171.455Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 144.519Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 199.358Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 164.597Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 219.264Âµs
>[negroni] Completed 200 OK in 137.893Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 150.877Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 200.432Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 204.065Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 144.579Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 169.772Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 196.835Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 254.321Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 123.392Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 145.724Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 4.629369ms
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 191.692Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 177.339Âµs
>[negroni] Completed 200 OK in 118.771Âµs
>[negroni] Started GET /clusters
>[negroni] Completed 200 OK in 267.888Âµs
>[negroni] Started DELETE /clusters/0a73c60efdd4673113b668afea101e6d
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 223.778Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 145.328Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 178.429Âµs
>[negroni] Completed 200 OK in 150.525Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 218.575Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 137.19Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 121.994Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 94.584Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 206.593Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 168.272Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 169.172Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 419.83Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 173.089Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 260.541Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 149.074Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 99.133Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 148.425Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 187.058Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 151.374Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 69.724Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 175.352Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 239.72Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 208.592Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 251.915Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 142.303Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 153.813Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 155.849Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 111.575Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 215.503Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 184.456Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 201.805Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 138.139Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 178.032Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 185.416Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 224.767Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 97.335Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 199.068Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 200.766Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 173.826Âµs
>[negroni] Completed 200 OK in 188.155Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 197.392Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 120.44Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 132.562Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 133.533Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 144.329Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 175.922Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 134.011Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 100.853Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 120.067Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 119.443Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 204.871Âµs
>[negroni] Completed 200 OK in 104.079Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 154.433Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 144.888Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 193.171Âµs
>[negroni] Completed 200 OK in 93.766Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 161.809Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 189.286Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 275.752Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 225.935Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 194.622Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 125.001Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 110.227Âµs
>[negroni] Completed 200 OK in 190.389Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 219.382Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 109.917Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 165.169Âµs
>[negroni] Completed 200 OK in 168.139Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 148.938Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 208.092Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 154.603Âµs
>[negroni] Completed 200 OK in 293.168Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 320.392Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 234.515Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 180.136Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 100.483Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 177.015Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 133.628Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 101.488Âµs
>[negroni] Completed 200 OK in 176.866Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 129.917Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 244.424Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 135.658Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 153.712Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 214.924Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 177.873Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 113.551Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 79.043Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 140.582Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 161.293Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 232.033Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 211.018Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 193.625Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 217.06Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 160.026Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 157.762Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 192.502Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 149.174Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 170.371Âµs
>[negroni] Completed 200 OK in 228.889Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 196.772Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 186.339Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 117.023Âµs
>[negroni] Completed 200 OK in 299.042Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 149.308Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 177.316Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 225.285Âµs
>[negroni] Completed 200 OK in 205.198Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 213.521Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 141.047Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 205.706Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 131.56Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 336.887Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 200.322Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 246.782Âµs
>[negroni] Completed 200 OK in 101.044Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 233.011Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 210.815Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 204.119Âµs
>[negroni] Completed 200 OK in 145.529Âµs
>[negroni] Started GET /clusters
>[negroni] Completed 200 OK in 309.708Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 118.982Âµs
>[negroni] Started GET /clusters
>[negroni] Completed 200 OK in 369.888Âµs
>[negroni] Started GET /clusters/0a73c60efdd4673113b668afea101e6d
>[negroni] Completed 200 OK in 1.534306ms
>[negroni] Started GET /nodes/278bd6b4e16a8e62ef15aaae22e6abc1
>[negroni] Completed 200 OK in 5.3361ms
>[negroni] Started GET /nodes/70423cc0bcd044fe5ba8bbbd256a3e49
>[negroni] Completed 200 OK in 3.529888ms
>[negroni] Started GET /nodes/d942fe6c0ee5691b7cc263968f97b650
>[negroni] Completed 200 OK in 2.998794ms
>[negroni] Started DELETE /nodes/278bd6b4e16a8e62ef15aaae22e6abc1
>[heketi] ERROR 2018/06/08 08:53:00 /src/github.com/heketi/heketi/apps/glusterfs/app_node.go:241: Unable to delete node [278bd6b4e16a8e62ef15aaae22e6abc1] because it contains devices
>[negroni] Completed 409 Conflict in 637.888Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 158.535Âµs
>[negroni] Started GET /clusters
>[negroni] Completed 200 OK in 161.091Âµs
>[negroni] Started GET /clusters/0a73c60efdd4673113b668afea101e6d
>[negroni] Completed 200 OK in 2.511085ms
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 189.939Âµs
>[negroni] Completed 200 OK in 91.001Âµs
>[negroni] Started POST /volumes
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 176.309Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 169.983Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 199.416Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 171.871Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 151.649Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 183.129Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 266.655Âµs
>[negroni] Completed 200 OK in 426.826Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 263.254Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 152.246Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 212.839Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 94.387Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 200.716Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 121.244Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 186.465Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 486.266Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 139.523Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 219.381Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 312.86Âµs
>[negroni] Completed 200 OK in 136.44Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 315.448Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 284.401Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 193.203Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 141.099Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 206.616Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 135.778Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 142.575Âµs
>[negroni] Completed 200 OK in 106.615Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 194.525Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 195.931Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 228.948Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 134.026Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 191.203Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 208.682Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 124.499Âµs
>[negroni] Completed 200 OK in 78.433Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 138.394Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 189.249Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 133.234Âµs
>[negroni] Completed 200 OK in 118.624Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 182.789Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 172.592Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 169.312Âµs
>[negroni] Completed 200 OK in 145.509Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 119.146Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 179.237Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 189.969Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 170.678Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 233.082Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 140.844Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 220.7Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 190.52Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 130.048Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 200.749Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 230.522Âµs
>[negroni] Completed 200 OK in 163.719Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 165.332Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 193.078Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 250.172Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 89.01Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 245.911Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 149.988Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 195.793Âµs
>[negroni] Completed 200 OK in 168.544Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 250.812Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 247.072Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 232.621Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 189.226Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 175.327Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 149.117Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 257.118Âµs
>[negroni] Completed 200 OK in 146.916Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 177.207Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 190.376Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 128.442Âµs
>[negroni] Completed 200 OK in 273.434Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 123.775Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 171.211Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 230.411Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 135.446Âµs
>[negroni] Started DELETE /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 193.545Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 114.323Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 139.633Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 117.512Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 146.006Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 149.994Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 307.202Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 196.432Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 173.762Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 199.894Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 192.882Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 151.749Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 185.651Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 243.171Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 191.621Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 170.986Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 176.919Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 216.835Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 187.129Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 140.782Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 154.249Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 259.465Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 230.488Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 129.562Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 216.058Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 250.272Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 279.697Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 181.568Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 181.268Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 266.399Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 188.951Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 176.348Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 189.142Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 222.362Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 220.728Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 199.605Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 189.656Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 199.576Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 284.005Âµs
>[negroni] Completed 200 OK in 98.077Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 142.811Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 127.572Âµs
>[negroni] Started GET /clusters
>[negroni] Completed 200 OK in 209.278Âµs
>[negroni] Started GET /clusters/0a73c60efdd4673113b668afea101e6d
>[negroni] Completed 200 OK in 1.576409ms
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 127.545Âµs
>[negroni] Completed 200 OK in 82.001Âµs
>[negroni] Started GET /nodes/278bd6b4e16a8e62ef15aaae22e6abc1
>[negroni] Completed 200 OK in 1.490437ms
>[negroni] Started GET /nodes/70423cc0bcd044fe5ba8bbbd256a3e49
>[negroni] Completed 200 OK in 1.849942ms
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 132.64Âµs
>[negroni] Started GET /nodes/d942fe6c0ee5691b7cc263968f97b650
>[negroni] Completed 200 OK in 1.681931ms
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 174.042Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 183.978Âµs
>[negroni] Completed 200 OK in 272Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 300.881Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 156.86Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 152.922Âµs
>[negroni] Completed 200 OK in 238.913Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 278.276Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 288.878Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 240.871Âµs
>[negroni] Completed 200 OK in 103.787Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 146.959Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 314.444Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 164.553Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 254.214Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 208.875Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 261.011Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 236.898Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 228.448Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 198.179Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 167.922Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 184.026Âµs
>[negroni] Completed 200 OK in 219.706Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 234.4Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 146.574Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 171.976Âµs
>[negroni] Completed 200 OK in 131.566Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 252.108Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 213.623Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 135.786Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 161.212Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 220.719Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 179.888Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 141.397Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 168.112Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 193.392Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 374.133Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 187.524Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 152.322Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 200 OK in 214.752Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 187.252Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 142.595Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 120.653Âµs
>[negroni] Started GET /queue/a8d46afb5176808836420be86f6fdcd0
>[negroni] Completed 401 Unauthorized in 135.43Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 252.895Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 236.851Âµs
>[negroni] Completed 200 OK in 127.913Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 204.956Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 136.291Âµs
>[negroni] Completed 200 OK in 124.726Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 139.038Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 237.525Âµs
>[negroni] Completed 200 OK in 154.932Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 162.292Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 208.122Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 203.585Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 186.485Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 172.828Âµs
>[negroni] Completed 200 OK in 175.176Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 249.86Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 143.579Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 92.984Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 204.612Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 143.516Âµs
>[negroni] Completed 200 OK in 124.198Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 213.661Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 190.928Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 153.579Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 213.106Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 186.802Âµs
>[negroni] Completed 200 OK in 143.239Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 192.886Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 201.125Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 134.249Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 165.416Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 278.384Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 182.418Âµs
>[kubeexec] ERROR 2018/06/08 08:53:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume create vol_9f9fb17746d0aad637b132875d2744e5 replica 3 10.70.46.122:/var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_cc2483d7b49bc029b5200a024cac7535/brick 10.70.47.76:/var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_2ed96343627983cf9667b7cee4052d17/brick 10.70.46.187:/var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_1bb49691eccf328025855a91ee8cbc66/brick] on glusterfs-storage-pg4xc: Err[command terminated with exit code 1]: Stdout [Error : Request timed out
>]: Stderr []
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 124.736Âµs
>[kubeexec] DEBUG 2018/06/08 08:53:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:42 UTC; 2h 30min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4900c968_6ae4_11e8_bc4b_005056a5f18a.slice/docker-438fb5fb16a82a3e701b28fa1f1ef8623dc37ca06374c604e5781a2b2ab2e06d.scope/system.slice/glusterd.service
>           ââ  433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1106 /usr/sbin/glusterfsd -s 10.70.46.122 --volfile-id heketidbstorage.10.70.46.122.var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.122-var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.pid -S /var/run/gluster/9fa3ac3b12826912287adc88c26ac134.socket --brick-name /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_29e5b36d921cca1b0699871d9727c932/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.log --xlator-option *-posix.glusterd-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ14514 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f8994cdb09351d83e70da08261d99c64.socket --xlator-option *replicate*.node-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0
>
>Jun 08 06:23:40 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:42 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 08:53:54 Periodic health check status: node 70423cc0bcd044fe5ba8bbbd256a3e49 up=true
>[cmdexec] INFO 2018/06/08 08:53:54 Check Glusterd service status in node dhcp47-76.lab.eng.blr.redhat.com
>[negroni] Completed 202 Accepted in 6m2.125589789s
>[asynchttp] INFO 2018/06/08 08:53:54 asynchttp.go:288: Started job 53ae2da32657329fa5a30bfda1414335
>[heketi] INFO 2018/06/08 08:53:54 Started async operation: Create Volume
>[heketi] INFO 2018/06/08 08:53:54 Creating brick 079040d8947a2eda08d5ddbdc41bfa83
>[heketi] INFO 2018/06/08 08:53:54 Creating brick ac396b39a38b350b3672e904f47f449e
>[heketi] INFO 2018/06/08 08:53:54 Creating brick 615bca09219fdb7b02ec4e0a6a62d4f7
>[heketi] INFO 2018/06/08 08:53:54 Allocating brick set #0
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 136.194Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 106.942Âµs
>[kubeexec] DEBUG 2018/06/08 08:53:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir -p /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_5a17b5c320a4f4efb036a3001a05544b
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:53:55 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_3a4297677881963e3f80124971d50eea/tp_5a17b5c320a4f4efb036a3001a05544b --virtualsize 2097152K --name brick_5a17b5c320a4f4efb036a3001a05544b
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_5a17b5c320a4f4efb036a3001a05544b" created.
>[kubeexec] DEBUG 2018/06/08 08:53:55 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_5a17b5c320a4f4efb036a3001a05544b
>Result: meta-data=/dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_5a17b5c320a4f4efb036a3001a05544b isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:53:55 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: awk "BEGIN {print \"/dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_5a17b5c320a4f4efb036a3001a05544b /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_5a17b5c320a4f4efb036a3001a05544b xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:53:55 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_5a17b5c320a4f4efb036a3001a05544b /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_5a17b5c320a4f4efb036a3001a05544b
>Result: 
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 99.509Âµs
>[kubeexec] DEBUG 2018/06/08 08:53:55 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_5a17b5c320a4f4efb036a3001a05544b/brick
>Result: 
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 149.642Âµs
>[negroni] Completed 200 OK in 95.895Âµs
>[kubeexec] DEBUG 2018/06/08 08:53:55 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: chown :2004 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_5a17b5c320a4f4efb036a3001a05544b/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:53:55 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: chmod 2775 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_5a17b5c320a4f4efb036a3001a05544b/brick
>Result: 
>[cmdexec] INFO 2018/06/08 08:53:55 Creating volume vol_bf31af76ef6c54e7e8f24f4d8711cb22 replica 3
>[kubeexec] ERROR 2018/06/08 08:53:56 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume stop vol_9f9fb17746d0aad637b132875d2744e5 force] on glusterfs-storage-pg4xc: Err[command terminated with exit code 1]: Stdout []: Stderr [volume stop: vol_9f9fb17746d0aad637b132875d2744e5: failed: Another transaction is in progress for vol_9f9fb17746d0aad637b132875d2744e5. Please try again after sometime.
>]
>[cmdexec] ERROR 2018/06/08 08:53:56 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:144: Unable to stop volume vol_9f9fb17746d0aad637b132875d2744e5: Unable to execute command on glusterfs-storage-pg4xc: volume stop: vol_9f9fb17746d0aad637b132875d2744e5: failed: Another transaction is in progress for vol_9f9fb17746d0aad637b132875d2744e5. Please try again after sometime.
>[kubeexec] DEBUG 2018/06/08 08:53:56 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir -p /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_ac396b39a38b350b3672e904f47f449e
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:53:56 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_3a4297677881963e3f80124971d50eea/tp_ac396b39a38b350b3672e904f47f449e --virtualsize 2097152K --name brick_ac396b39a38b350b3672e904f47f449e
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_ac396b39a38b350b3672e904f47f449e" created.
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 240.494Âµs
>[kubeexec] DEBUG 2018/06/08 08:53:56 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_ac396b39a38b350b3672e904f47f449e
>Result: meta-data=/dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_ac396b39a38b350b3672e904f47f449e isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 177.518Âµs
>[negroni] Completed 200 OK in 128.902Âµs
>[kubeexec] DEBUG 2018/06/08 08:53:56 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: awk "BEGIN {print \"/dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_ac396b39a38b350b3672e904f47f449e /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_ac396b39a38b350b3672e904f47f449e xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] ERROR 2018/06/08 08:53:56 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume create vol_b99532640a5201d243193159ee762ae4 replica 3 10.70.46.187:/var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_dad9ada287c446b0011af8dc964060e1/brick 10.70.46.122:/var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_3dbab6cd698d01ea7f00dbd81329643a/brick 10.70.47.76:/var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_df5559fdf15f6372e19e4e9f8bc1f129/brick] on glusterfs-storage-vsh2m: Err[command terminated with exit code 1]: Stdout [Error : Request timed out
>]: Stderr []
>[kubeexec] DEBUG 2018/06/08 08:53:56 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_ac396b39a38b350b3672e904f47f449e /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_ac396b39a38b350b3672e904f47f449e
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:53:57 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_ac396b39a38b350b3672e904f47f449e/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:53:57 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: chown :2005 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_ac396b39a38b350b3672e904f47f449e/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:53:57 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: chmod 2775 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_ac396b39a38b350b3672e904f47f449e/brick
>Result: 
>[kubeexec] ERROR 2018/06/08 08:53:57 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume delete vol_9f9fb17746d0aad637b132875d2744e5] on glusterfs-storage-pg4xc: Err[command terminated with exit code 1]: Stdout []: Stderr [volume delete: vol_9f9fb17746d0aad637b132875d2744e5: failed: Another transaction is in progress for vol_9f9fb17746d0aad637b132875d2744e5. Please try again after sometime.
>]
>[cmdexec] ERROR 2018/06/08 08:53:57 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:153: Unable to delete volume vol_9f9fb17746d0aad637b132875d2744e5: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_9f9fb17746d0aad637b132875d2744e5: failed: Another transaction is in progress for vol_9f9fb17746d0aad637b132875d2744e5. Please try again after sometime.
>[heketi] ERROR 2018/06/08 08:53:57 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:177: Error executing create volume: Unable to execute command on glusterfs-storage-pg4xc: 
>[heketi] WARNING 2018/06/08 08:53:57 Create Volume Exec requested retry
>[heketi] INFO 2018/06/08 08:53:57 Retry Create Volume (1)
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 180.286Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 170.771Âµs
>[negroni] Completed 200 OK in 88.561Âµs
>[kubeexec] ERROR 2018/06/08 08:53:58 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume stop vol_15e0122e942fc41f80666a3714670682 force] on glusterfs-storage-gxp7c: Err[command terminated with exit code 1]: Stdout []: Stderr [volume stop: vol_15e0122e942fc41f80666a3714670682: failed: Another transaction is in progress for vol_15e0122e942fc41f80666a3714670682. Please try again after sometime.
>]
>[cmdexec] ERROR 2018/06/08 08:53:58 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:144: Unable to stop volume vol_15e0122e942fc41f80666a3714670682: Unable to execute command on glusterfs-storage-gxp7c: volume stop: vol_15e0122e942fc41f80666a3714670682: failed: Another transaction is in progress for vol_15e0122e942fc41f80666a3714670682. Please try again after sometime.
>[kubeexec] DEBUG 2018/06/08 08:53:58 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 07:25:20 UTC; 1h 28min ago
>  Process: 828 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 830 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5cb2299_6aec_11e8_bc4b_005056a5f18a.slice/docker-7f9223abc9a8641cea68f374f4039235515978ee2dab970ac6cbef182c535426.scope/system.slice/glusterd.service
>           ââ  830 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1197 /usr/sbin/glusterfsd -s 10.70.47.76 --volfile-id heketidbstorage.10.70.47.76.var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.76-var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.pid -S /var/run/gluster/b6976f9ddc3abc90e1dfa91c35a7c611.socket --brick-name /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_54c06cdf52c6fe1e3a3f8a9d39cde33d/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.log --xlator-option *-posix.glusterd-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ13533 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/ec468453b11caed5259f5e1abcc65b44.socket --xlator-option *replicate*.node-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94
>           ââ13534 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/ec468453b11caed5259f5e1abcc65b44.socket --xlator-option *replicate*.node-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94
>           ââ13541 /usr/bin/python /var/lib/glusterd/hooks/1/delete/post/S57glusterfind-delete-post --volname=vol_9c85de66c12db0a72b4d16fe888ff74d
>           ââ13543 gluster system:: getwd
>           ââ13545 /usr/bin/python /usr/libexec/glusterfs/python/syncdaemon/gsyncd.py --version
>
>Jun 08 07:25:17 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 07:25:20 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Jun 08 08:53:58 dhcp47-76.lab.eng.blr.redhat.com glusterd[830]: [2018-06-08 08:53:58.338441] C [rpc-clnt.c:465:rpc_clnt_fill_request_info] 0-management: cannot lookup the saved frame corresponding to xid (103)
>[heketi] INFO 2018/06/08 08:53:58 Periodic health check status: node d942fe6c0ee5691b7cc263968f97b650 up=true
>[heketi] INFO 2018/06/08 08:53:58 Cleaned 0 nodes from health cache
>[heketi] INFO 2018/06/08 08:53:58 Starting Node Health Status refresh
>[cmdexec] INFO 2018/06/08 08:53:58 Check Glusterd service status in node dhcp46-187.lab.eng.blr.redhat.com
>[negroni] Started GET /clusters
>[negroni] Completed 200 OK in 185.659Âµs
>[negroni] Started GET /clusters/0a73c60efdd4673113b668afea101e6d
>[negroni] Completed 200 OK in 2.55109ms
>[negroni] Completed 202 Accepted in 6m5.812084946s
>[asynchttp] INFO 2018/06/08 08:53:58 asynchttp.go:288: Started job c2300af895526bf82435fd82aaea0ce8
>[heketi] INFO 2018/06/08 08:53:58 Started async operation: Create Volume
>[heketi] INFO 2018/06/08 08:53:58 Creating brick 35ab4a65e06fcf26cf320c522f9afc92
>[heketi] INFO 2018/06/08 08:53:58 Creating brick 6f7534e6df0ad44d669269ef560a18ed
>[heketi] INFO 2018/06/08 08:53:58 Creating brick e0e48c633e08f573cc3b70b99f47faea
>[heketi] INFO 2018/06/08 08:53:58 Allocating brick set #0
>[kubeexec] DEBUG 2018/06/08 08:53:58 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir -p /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_615bca09219fdb7b02ec4e0a6a62d4f7
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:53:58 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir -p /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_35ab4a65e06fcf26cf320c522f9afc92
>Result: 
>[negroni] Started GET /nodes/278bd6b4e16a8e62ef15aaae22e6abc1
>[negroni] Completed 200 OK in 9.444592ms
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 125.728Âµs
>[negroni] Started GET /nodes/70423cc0bcd044fe5ba8bbbd256a3e49
>[negroni] Completed 200 OK in 2.774778ms
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 117.279Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 97.881Âµs
>[kubeexec] DEBUG 2018/06/08 08:53:58 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_9394bc70699b006c5460c9f654cf345f/tp_615bca09219fdb7b02ec4e0a6a62d4f7 --virtualsize 2097152K --name brick_615bca09219fdb7b02ec4e0a6a62d4f7
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_615bca09219fdb7b02ec4e0a6a62d4f7" created.
>[negroni] Started GET /nodes/d942fe6c0ee5691b7cc263968f97b650
>[negroni] Completed 200 OK in 1.50699ms
>[kubeexec] DEBUG 2018/06/08 08:53:58 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_3a4297677881963e3f80124971d50eea/tp_35ab4a65e06fcf26cf320c522f9afc92 --virtualsize 2097152K --name brick_35ab4a65e06fcf26cf320c522f9afc92
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_35ab4a65e06fcf26cf320c522f9afc92" created.
>[kubeexec] DEBUG 2018/06/08 08:53:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_615bca09219fdb7b02ec4e0a6a62d4f7
>Result: meta-data=/dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_615bca09219fdb7b02ec4e0a6a62d4f7 isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:53:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_35ab4a65e06fcf26cf320c522f9afc92
>Result: meta-data=/dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_35ab4a65e06fcf26cf320c522f9afc92 isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:53:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: awk "BEGIN {print \"/dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_615bca09219fdb7b02ec4e0a6a62d4f7 /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_615bca09219fdb7b02ec4e0a6a62d4f7 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:53:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: awk "BEGIN {print \"/dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_35ab4a65e06fcf26cf320c522f9afc92 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_35ab4a65e06fcf26cf320c522f9afc92 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:53:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_615bca09219fdb7b02ec4e0a6a62d4f7 /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_615bca09219fdb7b02ec4e0a6a62d4f7
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:53:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_35ab4a65e06fcf26cf320c522f9afc92 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_35ab4a65e06fcf26cf320c522f9afc92
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:53:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_615bca09219fdb7b02ec4e0a6a62d4f7/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:53:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_35ab4a65e06fcf26cf320c522f9afc92/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:53:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: chown :2005 /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_615bca09219fdb7b02ec4e0a6a62d4f7/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:53:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: chown :2006 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_35ab4a65e06fcf26cf320c522f9afc92/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:53:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: chmod 2775 /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_615bca09219fdb7b02ec4e0a6a62d4f7/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:53:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: chmod 2775 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_35ab4a65e06fcf26cf320c522f9afc92/brick
>Result: 
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 123.075Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 116.861Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 59.127Âµs
>[heketi] WARNING 2018/06/08 08:54:00 Create Volume Exec requested retry
>[heketi] INFO 2018/06/08 08:54:00 Retry Create Volume (1)
>[kubeexec] ERROR 2018/06/08 08:54:00 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume delete vol_ad1e5849e9566f1bcaa09cfb9c0b96ef] on glusterfs-storage-vsh2m: Err[command terminated with exit code 1]: Stdout []: Stderr [volume delete: vol_ad1e5849e9566f1bcaa09cfb9c0b96ef: failed: Locking failed on 10.70.47.76. Please check log file for details.
>]
>[cmdexec] ERROR 2018/06/08 08:54:00 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:153: Unable to delete volume vol_ad1e5849e9566f1bcaa09cfb9c0b96ef: Unable to execute command on glusterfs-storage-vsh2m: volume delete: vol_ad1e5849e9566f1bcaa09cfb9c0b96ef: failed: Locking failed on 10.70.47.76. Please check log file for details.
>[heketi] ERROR 2018/06/08 08:54:00 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:177: Error executing create volume: Unable to execute command on glusterfs-storage-vsh2m: 
>[kubeexec] DEBUG 2018/06/08 08:54:00 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir -p /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_079040d8947a2eda08d5ddbdc41bfa83
>Result: 
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 125.764Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 110.675Âµs
>[negroni] Completed 200 OK in 72.483Âµs
>[kubeexec] DEBUG 2018/06/08 08:54:00 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_b861af9cc67f5bf8d5a8f9e38f56b14e/tp_079040d8947a2eda08d5ddbdc41bfa83 --virtualsize 2097152K --name brick_079040d8947a2eda08d5ddbdc41bfa83
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_079040d8947a2eda08d5ddbdc41bfa83" created.
>[kubeexec] DEBUG 2018/06/08 08:54:01 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_079040d8947a2eda08d5ddbdc41bfa83
>Result: meta-data=/dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_079040d8947a2eda08d5ddbdc41bfa83 isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:54:01 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: awk "BEGIN {print \"/dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_079040d8947a2eda08d5ddbdc41bfa83 /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_079040d8947a2eda08d5ddbdc41bfa83 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:01 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_079040d8947a2eda08d5ddbdc41bfa83 /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_079040d8947a2eda08d5ddbdc41bfa83
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:01 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_079040d8947a2eda08d5ddbdc41bfa83/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:01 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: chown :2005 /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_079040d8947a2eda08d5ddbdc41bfa83/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:01 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: chmod 2775 /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_079040d8947a2eda08d5ddbdc41bfa83/brick
>Result: 
>[cmdexec] INFO 2018/06/08 08:54:01 Creating volume vol_cc8a686464bb4017c91ba7294ee1b091 replica 3
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 168.411Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 223.931Âµs
>[negroni] Completed 200 OK in 121.67Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 125.467Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 122.242Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 301.545Âµs
>[kubeexec] ERROR 2018/06/08 08:54:03 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume stop vol_b99532640a5201d243193159ee762ae4 force] on glusterfs-storage-vsh2m: Err[command terminated with exit code 1]: Stdout []: Stderr [volume stop: vol_b99532640a5201d243193159ee762ae4: failed: Another transaction is in progress for vol_b99532640a5201d243193159ee762ae4. Please try again after sometime.
>]
>[cmdexec] ERROR 2018/06/08 08:54:03 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:144: Unable to stop volume vol_b99532640a5201d243193159ee762ae4: Unable to execute command on glusterfs-storage-vsh2m: volume stop: vol_b99532640a5201d243193159ee762ae4: failed: Another transaction is in progress for vol_b99532640a5201d243193159ee762ae4. Please try again after sometime.
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 115.044Âµs
>[kubeexec] DEBUG 2018/06/08 08:54:03 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script volume create vol_bf31af76ef6c54e7e8f24f4d8711cb22 replica 3 10.70.47.76:/var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_c6935061d1c9dbb05f316e3d87080a38/brick 10.70.46.187:/var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_f4fd75475848eb3cf3c5f985495d64f3/brick 10.70.46.122:/var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_5a17b5c320a4f4efb036a3001a05544b/brick
>Result: volume create: vol_bf31af76ef6c54e7e8f24f4d8711cb22: success: please start the volume to access data
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 110.137Âµs
>[negroni] Completed 200 OK in 75.373Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 118.054Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 106.669Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 67.57Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 159.813Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 212.159Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 110.882Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 197.856Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 265.108Âµs
>[negroni] Completed 200 OK in 302.648Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 122.731Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 121.474Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 200 OK in 93.195Âµs
>[kubeexec] DEBUG 2018/06/08 08:54:07 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: gluster --mode=script volume stop vol_9f9fb17746d0aad637b132875d2744e5 force
>Result: volume stop: vol_9f9fb17746d0aad637b132875d2744e5: success
>[kubeexec] DEBUG 2018/06/08 08:54:08 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:33 UTC; 2h 30min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48fed567_6ae4_11e8_bc4b_005056a5f18a.slice/docker-a506d9aaca4b440d184df40d01e90b2040811c8090bf4c19d892081307b28c0c.scope/system.slice/glusterd.service
>           ââ  433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1228 /usr/sbin/glusterfsd -s 10.70.46.187 --volfile-id heketidbstorage.10.70.46.187.var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.187-var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.pid -S /var/run/gluster/c08898c2e3b0190c62df3a9d4a013c92.socket --brick-name /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.log --xlator-option *-posix.glusterd-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ15453 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/8fb729a0bbd51b204491c603c84acaa4.socket --xlator-option *replicate*.node-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034
>           ââ15454 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/8fb729a0bbd51b204491c603c84acaa4.socket --xlator-option *replicate*.node-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034
>
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Unit glusterd.service entered failed state.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: glusterd.service failed.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:33 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 08:54:08 Periodic health check status: node 278bd6b4e16a8e62ef15aaae22e6abc1 up=true
>[cmdexec] INFO 2018/06/08 08:54:08 Check Glusterd service status in node dhcp46-122.lab.eng.blr.redhat.com
>[negroni] Completed 202 Accepted in 6m15.226372505s
>[asynchttp] INFO 2018/06/08 08:54:08 asynchttp.go:288: Started job 252b1a3974a7d2a66ea36f37f2c82074
>[heketi] INFO 2018/06/08 08:54:08 Started async operation: Create Volume
>[heketi] INFO 2018/06/08 08:54:08 Creating brick 063b0d12615d3375614df036642dfa39
>[heketi] INFO 2018/06/08 08:54:08 Creating brick 6bfb8b3bea31f51d5be26acc7e81ce9a
>[heketi] INFO 2018/06/08 08:54:08 Creating brick 4dc091ed7622300b60597d7a49fae798
>[negroni] Completed 202 Accepted in 6m2.241897404s
>[asynchttp] INFO 2018/06/08 08:54:08 asynchttp.go:288: Started job 20e932645b6c4b3853e7a46917e686e7
>[heketi] INFO 2018/06/08 08:54:08 Started async operation: Delete Volume
>[asynchttp] INFO 2018/06/08 08:54:08 asynchttp.go:292: Completed job b7642ac5823fb25c80d83504800b9c18 in 8m2.242006294s
>[heketi] ERROR 2018/06/08 08:54:08 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:1185: Delete Volume Failed: Unable to determine snapshot information from volume vol_9c85de66c12db0a72b4d16fe888ff74d: EOF
>[heketi] INFO 2018/06/08 08:54:08 Allocating brick set #0
>[kubeexec] DEBUG 2018/06/08 08:54:08 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:42 UTC; 2h 30min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4900c968_6ae4_11e8_bc4b_005056a5f18a.slice/docker-438fb5fb16a82a3e701b28fa1f1ef8623dc37ca06374c604e5781a2b2ab2e06d.scope/system.slice/glusterd.service
>           ââ  433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1106 /usr/sbin/glusterfsd -s 10.70.46.122 --volfile-id heketidbstorage.10.70.46.122.var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.122-var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.pid -S /var/run/gluster/9fa3ac3b12826912287adc88c26ac134.socket --brick-name /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_29e5b36d921cca1b0699871d9727c932/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.log --xlator-option *-posix.glusterd-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ15640 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f8994cdb09351d83e70da08261d99c64.socket --xlator-option *replicate*.node-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0
>           ââ15641 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f8994cdb09351d83e70da08261d99c64.socket --xlator-option *replicate*.node-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0
>
>Jun 08 06:23:40 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:42 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 08:54:08 Periodic health check status: node 70423cc0bcd044fe5ba8bbbd256a3e49 up=true
>[cmdexec] INFO 2018/06/08 08:54:08 Check Glusterd service status in node dhcp47-76.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 08:54:08 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir -p /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_6f7534e6df0ad44d669269ef560a18ed
>Result: 
>[negroni] Completed 202 Accepted in 5m32.357531985s
>[asynchttp] INFO 2018/06/08 08:54:08 asynchttp.go:288: Started job d5734675939ff78001cf479cb6e9b97c
>[heketi] INFO 2018/06/08 08:54:08 Started async operation: Create Volume
>[heketi] INFO 2018/06/08 08:54:08 Creating brick cb9dab914105ca3d691abed1d53d7df9
>[heketi] INFO 2018/06/08 08:54:08 Creating brick cd0ffc6bd88bdb1d969dceaafda517d8
>[heketi] INFO 2018/06/08 08:54:08 Creating brick 47b16a0d2bab878899e523ab33a0c258
>[heketi] INFO 2018/06/08 08:54:08 Allocating brick set #0
>[kubeexec] DEBUG 2018/06/08 08:54:08 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir -p /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_063b0d12615d3375614df036642dfa39
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:08 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_d389f0278a774bd7443a09af960961d8/tp_6f7534e6df0ad44d669269ef560a18ed --virtualsize 2097152K --name brick_6f7534e6df0ad44d669269ef560a18ed
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_6f7534e6df0ad44d669269ef560a18ed" created.
>[kubeexec] DEBUG 2018/06/08 08:54:08 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_6f7534e6df0ad44d669269ef560a18ed
>Result: meta-data=/dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_6f7534e6df0ad44d669269ef560a18ed isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:54:08 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_66a9af9f6bac95f9d8d556a2f14c29d3/tp_063b0d12615d3375614df036642dfa39 --virtualsize 2097152K --name brick_063b0d12615d3375614df036642dfa39
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_063b0d12615d3375614df036642dfa39" created.
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 148.067Âµs
>[kubeexec] DEBUG 2018/06/08 08:54:08 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: awk "BEGIN {print \"/dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_6f7534e6df0ad44d669269ef560a18ed /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_6f7534e6df0ad44d669269ef560a18ed xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 182.59Âµs
>[negroni] Started GET /queue/b7642ac5823fb25c80d83504800b9c18
>[negroni] Completed 500 Internal Server Error in 129.978Âµs
>[kubeexec] DEBUG 2018/06/08 08:54:08 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_063b0d12615d3375614df036642dfa39
>Result: meta-data=/dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_063b0d12615d3375614df036642dfa39 isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:54:08 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: awk "BEGIN {print \"/dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_063b0d12615d3375614df036642dfa39 /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_063b0d12615d3375614df036642dfa39 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:08 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_6f7534e6df0ad44d669269ef560a18ed /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_6f7534e6df0ad44d669269ef560a18ed
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:09 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_6f7534e6df0ad44d669269ef560a18ed/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:09 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_063b0d12615d3375614df036642dfa39 /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_063b0d12615d3375614df036642dfa39
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:09 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: chown :2006 /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_6f7534e6df0ad44d669269ef560a18ed/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:09 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_063b0d12615d3375614df036642dfa39/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:09 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: chmod 2775 /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_6f7534e6df0ad44d669269ef560a18ed/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:09 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: chown :2007 /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_063b0d12615d3375614df036642dfa39/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:09 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: chmod 2775 /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_063b0d12615d3375614df036642dfa39/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:09 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script snapshot list vol_5db55c483f6984d9917cfe4b3c8b3cbc --xml
>Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
><cliOutput>
>  <opRet>0</opRet>
>  <opErrno>0</opErrno>
>  <opErrstr/>
>  <snapList>
>    <count>0</count>
>  </snapList>
></cliOutput>
>[kubeexec] DEBUG 2018/06/08 08:54:09 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir -p /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_47b16a0d2bab878899e523ab33a0c258
>Result: 
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 100.983Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 118.199Âµs
>[kubeexec] DEBUG 2018/06/08 08:54:10 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_3a4297677881963e3f80124971d50eea/tp_47b16a0d2bab878899e523ab33a0c258 --virtualsize 2097152K --name brick_47b16a0d2bab878899e523ab33a0c258
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_47b16a0d2bab878899e523ab33a0c258" created.
>[kubeexec] DEBUG 2018/06/08 08:54:10 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_47b16a0d2bab878899e523ab33a0c258
>Result: meta-data=/dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_47b16a0d2bab878899e523ab33a0c258 isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:54:10 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: awk "BEGIN {print \"/dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_47b16a0d2bab878899e523ab33a0c258 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_47b16a0d2bab878899e523ab33a0c258 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:10 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_47b16a0d2bab878899e523ab33a0c258 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_47b16a0d2bab878899e523ab33a0c258
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:10 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_47b16a0d2bab878899e523ab33a0c258/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:10 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: chown :2001 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_47b16a0d2bab878899e523ab33a0c258/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:10 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: chmod 2775 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_47b16a0d2bab878899e523ab33a0c258/brick
>Result: 
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 131.883Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 106.242Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 130.844Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 184.878Âµs
>[kubeexec] DEBUG 2018/06/08 08:54:12 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script volume start vol_bf31af76ef6c54e7e8f24f4d8711cb22
>Result: volume start: vol_bf31af76ef6c54e7e8f24f4d8711cb22: success
>[kubeexec] DEBUG 2018/06/08 08:54:12 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: gluster --mode=script volume create vol_cc8a686464bb4017c91ba7294ee1b091 replica 3 10.70.46.187:/var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_079040d8947a2eda08d5ddbdc41bfa83/brick 10.70.46.122:/var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_ac396b39a38b350b3672e904f47f449e/brick 10.70.47.76:/var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_615bca09219fdb7b02ec4e0a6a62d4f7/brick
>Result: volume create: vol_cc8a686464bb4017c91ba7294ee1b091: success: please start the volume to access data
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 139.314Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 252.798Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 142.751Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 135.588Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 173.966Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 136.43Âµs
>[kubeexec] DEBUG 2018/06/08 08:54:15 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script volume stop vol_5db55c483f6984d9917cfe4b3c8b3cbc force
>Result: volume stop: vol_5db55c483f6984d9917cfe4b3c8b3cbc: success
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 148.256Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 155.035Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 140.326Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 155.409Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 146.566Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 192.778Âµs
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 148.335Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 202.201Âµs
>[kubeexec] DEBUG 2018/06/08 08:54:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script volume delete vol_15e0122e942fc41f80666a3714670682
>Result: volume delete: vol_15e0122e942fc41f80666a3714670682: success
>[heketi] WARNING 2018/06/08 08:54:19 Create Volume Exec requested retry
>[heketi] INFO 2018/06/08 08:54:19 Retry Create Volume (1)
>[heketi] ERROR 2018/06/08 08:54:19 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:177: Error executing create volume: Unable to execute command on glusterfs-storage-gxp7c: 
>[kubeexec] DEBUG 2018/06/08 08:54:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: gluster --mode=script volume start vol_cc8a686464bb4017c91ba7294ee1b091
>Result: volume start: vol_cc8a686464bb4017c91ba7294ee1b091: success
>[kubeexec] DEBUG 2018/06/08 08:54:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script volume delete vol_5db55c483f6984d9917cfe4b3c8b3cbc
>Result: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: success
>[heketi] INFO 2018/06/08 08:54:19 Deleting brick 214eb9006f9103530a1d0310d5f5dcfc
>[heketi] INFO 2018/06/08 08:54:19 Deleting brick 4f4d753d298c99eac492c32006c74484
>[heketi] INFO 2018/06/08 08:54:19 Deleting brick f05028077b974ebc1f6621aee2184169
>[kubeexec] DEBUG 2018/06/08 08:54:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir -p /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_e0e48c633e08f573cc3b70b99f47faea
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mount | grep -w /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_214eb9006f9103530a1d0310d5f5dcfc | cut -d" " -f1
>Result: /dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_214eb9006f9103530a1d0310d5f5dcfc
>[heketi] ERROR 2018/06/08 08:54:19 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:177: Error executing create volume: Unable to execute command on glusterfs-storage-vsh2m: 
>[kubeexec] DEBUG 2018/06/08 08:54:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: gluster --mode=script volume delete vol_b99532640a5201d243193159ee762ae4
>Result: volume delete: vol_b99532640a5201d243193159ee762ae4: success
>[heketi] WARNING 2018/06/08 08:54:19 Create Volume Exec requested retry
>[heketi] INFO 2018/06/08 08:54:19 Retry Create Volume (1)
>[kubeexec] DEBUG 2018/06/08 08:54:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_214eb9006f9103530a1d0310d5f5dcfc
>
>Result:   vg_66a9af9f6bac95f9d8d556a2f14c29d3/tp_214eb9006f9103530a1d0310d5f5dcfc
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 172.747Âµs
>[kubeexec] DEBUG 2018/06/08 08:54:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: umount /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_214eb9006f9103530a1d0310d5f5dcfc
>Result: 
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 117.711Âµs
>[kubeexec] DEBUG 2018/06/08 08:54:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: gluster --mode=script volume delete vol_9f9fb17746d0aad637b132875d2744e5
>Result: volume delete: vol_9f9fb17746d0aad637b132875d2744e5: success
>[heketi] INFO 2018/06/08 08:54:19 Deleting brick cc2483d7b49bc029b5200a024cac7535
>[heketi] INFO 2018/06/08 08:54:19 Deleting brick 2ed96343627983cf9667b7cee4052d17
>[heketi] INFO 2018/06/08 08:54:19 Deleting brick 1bb49691eccf328025855a91ee8cbc66
>[kubeexec] DEBUG 2018/06/08 08:54:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_9394bc70699b006c5460c9f654cf345f/tp_e0e48c633e08f573cc3b70b99f47faea --virtualsize 2097152K --name brick_e0e48c633e08f573cc3b70b99f47faea
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_e0e48c633e08f573cc3b70b99f47faea" created.
>[kubeexec] DEBUG 2018/06/08 08:54:20 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_e0e48c633e08f573cc3b70b99f47faea
>Result: meta-data=/dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_e0e48c633e08f573cc3b70b99f47faea isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:54:20 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: awk "BEGIN {print \"/dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_e0e48c633e08f573cc3b70b99f47faea /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_e0e48c633e08f573cc3b70b99f47faea xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:20 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_e0e48c633e08f573cc3b70b99f47faea /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_e0e48c633e08f573cc3b70b99f47faea
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:20 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: sed -i.save "/brick_214eb9006f9103530a1d0310d5f5dcfc/d" /var/lib/heketi/fstab
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:20 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_e0e48c633e08f573cc3b70b99f47faea/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:20 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: chown :2006 /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_e0e48c633e08f573cc3b70b99f47faea/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:20 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: chmod 2775 /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_e0e48c633e08f573cc3b70b99f47faea/brick
>Result: 
>[cmdexec] INFO 2018/06/08 08:54:20 Creating volume vol_2bf097c60bd8b38bfcb4327727ca5681 replica 3
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 115.086Âµs
>[negroni] Started DELETE /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 152.277Âµs
>[kubeexec] DEBUG 2018/06/08 08:54:20 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mount | grep -w /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_cc2483d7b49bc029b5200a024cac7535 | cut -d" " -f1
>Result: /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_cc2483d7b49bc029b5200a024cac7535
>[kubeexec] DEBUG 2018/06/08 08:54:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir -p /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_4dc091ed7622300b60597d7a49fae798
>Result: 
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 158.285Âµs
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 109.615Âµs
>[kubeexec] DEBUG 2018/06/08 08:54:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvremove --autobackup=n -f /dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_214eb9006f9103530a1d0310d5f5dcfc
>
>Result:   Logical volume "brick_214eb9006f9103530a1d0310d5f5dcfc" successfully removed
>[kubeexec] DEBUG 2018/06/08 08:54:22 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_96f1667f2f1ced2c5ef94772922be93b/tp_4dc091ed7622300b60597d7a49fae798 --virtualsize 2097152K --name brick_4dc091ed7622300b60597d7a49fae798
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_4dc091ed7622300b60597d7a49fae798" created.
>[kubeexec] DEBUG 2018/06/08 08:54:22 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_4dc091ed7622300b60597d7a49fae798
>Result: meta-data=/dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_4dc091ed7622300b60597d7a49fae798 isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:54:22 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: awk "BEGIN {print \"/dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_4dc091ed7622300b60597d7a49fae798 /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_4dc091ed7622300b60597d7a49fae798 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:22 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_4dc091ed7622300b60597d7a49fae798 /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_4dc091ed7622300b60597d7a49fae798
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:22 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_4dc091ed7622300b60597d7a49fae798/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:22 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: chown :2007 /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_4dc091ed7622300b60597d7a49fae798/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:22 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: chmod 2775 /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_4dc091ed7622300b60597d7a49fae798/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:22 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 07:25:20 UTC; 1h 29min ago
>  Process: 828 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 830 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5cb2299_6aec_11e8_bc4b_005056a5f18a.slice/docker-7f9223abc9a8641cea68f374f4039235515978ee2dab970ac6cbef182c535426.scope/system.slice/glusterd.service
>           ââ  830 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1197 /usr/sbin/glusterfsd -s 10.70.47.76 --volfile-id heketidbstorage.10.70.47.76.var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.76-var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.pid -S /var/run/gluster/b6976f9ddc3abc90e1dfa91c35a7c611.socket --brick-name /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_54c06cdf52c6fe1e3a3f8a9d39cde33d/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.log --xlator-option *-posix.glusterd-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ13734 /usr/sbin/glusterfsd -s 10.70.47.76 --volfile-id vol_bf31af76ef6c54e7e8f24f4d8711cb22.10.70.47.76.var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_c6935061d1c9dbb05f316e3d87080a38-brick -p /var/run/gluster/vols/vol_bf31af76ef6c54e7e8f24f4d8711cb22/10.70.47.76-var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_c6935061d1c9dbb05f316e3d87080a38-brick.pid -S /var/run/gluster/7aabe8187d4b998d498dd5d4edbd2626.socket --brick-name /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_c6935061d1c9dbb05f316e3d87080a38/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_c6935061d1c9dbb05f316e3d87080a38-brick.log --xlator-option *-posix.glusterd-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94 --brick-port 49153 --xlator-option vol_bf31af76ef6c54e7e8f24f4d8711cb22-server.listen-port=49153
>           ââ13858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/ec468453b11caed5259f5e1abcc65b44.socket --xlator-option *replicate*.node-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94
>
>Jun 08 07:25:17 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 07:25:20 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Jun 08 08:53:58 dhcp47-76.lab.eng.blr.redhat.com glusterd[830]: [2018-06-08 08:53:58.338441] C [rpc-clnt.c:465:rpc_clnt_fill_request_info] 0-management: cannot lookup the saved frame corresponding to xid (103)
>[heketi] INFO 2018/06/08 08:54:22 Periodic health check status: node d942fe6c0ee5691b7cc263968f97b650 up=true
>[heketi] INFO 2018/06/08 08:54:22 Cleaned 0 nodes from health cache
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 200 OK in 131.894Âµs
>[negroni] Completed 202 Accepted in 5m46.911880344s
>[asynchttp] INFO 2018/06/08 08:54:22 asynchttp.go:288: Started job 086aebe6792ba08c072791f3d9745777
>[heketi] INFO 2018/06/08 08:54:22 Started async operation: Create Volume
>[heketi] INFO 2018/06/08 08:54:22 Creating brick c4423da2a93ab3c5917479262cf1d93a
>[heketi] INFO 2018/06/08 08:54:22 Creating brick 91d54591c1b07bc3a93d925cfbbd0f23
>[heketi] INFO 2018/06/08 08:54:22 Creating brick e3ebda9db9a085114fe9732ab2df4869
>[heketi] INFO 2018/06/08 08:54:22 Allocating brick set #0
>[negroni] Completed 202 Accepted in 5m46.981032141s
>[asynchttp] INFO 2018/06/08 08:54:22 asynchttp.go:288: Started job 12c7a0bf9d0844701651eb57cc4883ef
>[heketi] INFO 2018/06/08 08:54:22 Started async operation: Create Volume
>[heketi] INFO 2018/06/08 08:54:22 Creating brick aec48b6cbce6be7256a2b86efac6aef6
>[heketi] INFO 2018/06/08 08:54:22 Creating brick abd8495bfd886d541a449967f7bf70c0
>[heketi] INFO 2018/06/08 08:54:22 Creating brick 26379a9db79930ad00a37453b492ebac
>[heketi] INFO 2018/06/08 08:54:22 Allocating brick set #0
>[negroni] Completed 202 Accepted in 5m47.004154685s
>[asynchttp] INFO 2018/06/08 08:54:22 asynchttp.go:288: Started job 7e56f206e49c37c88b54444c74b73010
>[heketi] INFO 2018/06/08 08:54:22 Started async operation: Create Volume
>[heketi] INFO 2018/06/08 08:54:22 Creating brick 24c8f78999c94f4eacb087ac1fc563d1
>[heketi] INFO 2018/06/08 08:54:22 Creating brick c77746b3040810cf2ae10fc585c67fd5
>[heketi] INFO 2018/06/08 08:54:22 Creating brick 6c8f55013bee289cd516ef97746721ff
>[heketi] INFO 2018/06/08 08:54:22 Allocating brick set #0
>[negroni] Completed 202 Accepted in 5m47.023976505s
>[asynchttp] INFO 2018/06/08 08:54:22 asynchttp.go:288: Started job 963dc2b509a04cdd5db1a8531a35c01c
>[heketi] INFO 2018/06/08 08:54:22 Started async operation: Create Volume
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 200 OK in 87.904Âµs
>[heketi] INFO 2018/06/08 08:54:22 Creating brick cdc8c0d24f68026f06c17385d2fd0029
>[heketi] INFO 2018/06/08 08:54:22 Creating brick 9be3d8f4748b4c09541f8a9de90c4c34
>[heketi] INFO 2018/06/08 08:54:22 Creating brick cecca27e94262fbaae81106a8cfeb0b2
>[heketi] INFO 2018/06/08 08:54:22 Allocating brick set #0
>[negroni] Completed 202 Accepted in 5m47.043182703s
>[asynchttp] INFO 2018/06/08 08:54:22 asynchttp.go:288: Started job d4f0d63988cfe91663765df353eac613
>[heketi] INFO 2018/06/08 08:54:22 Started async operation: Create Volume
>[heketi] INFO 2018/06/08 08:54:22 Creating brick d985a824b73813069566b0a71bb73269
>[heketi] INFO 2018/06/08 08:54:22 Creating brick dddf81e8d43f425cc26134166b21f15e
>[heketi] INFO 2018/06/08 08:54:22 Creating brick ac27bcfa769ceba759b678d9e6b751cb
>[heketi] INFO 2018/06/08 08:54:22 Allocating brick set #0
>[negroni] Completed 202 Accepted in 5m47.062216544s
>[asynchttp] INFO 2018/06/08 08:54:22 asynchttp.go:288: Started job 0f488f564115ab81bd476de5a18002e4
>[heketi] INFO 2018/06/08 08:54:22 Started async operation: Create Volume
>[heketi] ERROR 2018/06/08 08:54:22 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:399: Pending brick 214eb9006f9103530a1d0310d5f5dcfc can not be deleted
>[heketi] INFO 2018/06/08 08:54:22 Creating brick d8a382627ef730cbe0664d9439fd6a59
>[heketi] INFO 2018/06/08 08:54:22 Creating brick bd4fcfec1db80add56b57e18bd681378
>[heketi] ERROR 2018/06/08 08:54:22 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:1167: Delete Volume Build Failed: The target exists, contains other items, or is in use.
>[heketi] INFO 2018/06/08 08:54:22 Creating brick 0a07a24688dfe13d4e1726b8d05e24a7
>[negroni] Completed 500 Internal Server Error in 5m32.077688655s
>[asynchttp] INFO 2018/06/08 08:54:22 asynchttp.go:292: Completed job 548c6fbe203bb93f9a4ecb56f976926d in 7m32.07842258s
>[heketi] ERROR 2018/06/08 08:54:22 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:1185: Delete Volume Failed: Unable to delete volume vol_a4a6e4892da299f6c5634b8a2def697e: Unable to execute command on glusterfs-storage-vsh2m: volume delete: vol_a4a6e4892da299f6c5634b8a2def697e: failed: Another transaction is in progress for vol_a4a6e4892da299f6c5634b8a2def697e. Please try again after sometime.
>[heketi] INFO 2018/06/08 08:54:22 Allocating brick set #0
>[negroni] Completed 202 Accepted in 5m2.105699533s
>[asynchttp] INFO 2018/06/08 08:54:22 asynchttp.go:288: Started job 44bb2a0e030ba450c0c64afe4b039c77
>[heketi] INFO 2018/06/08 08:54:22 Started async operation: Create Volume
>[heketi] INFO 2018/06/08 08:54:22 Creating brick 8d39164e3ebba9d10c8b6ed33dafabc7
>[heketi] INFO 2018/06/08 08:54:22 Creating brick c740e50db586c869e27205aee2820dd7
>[heketi] INFO 2018/06/08 08:54:22 Creating brick f88d16c3bc0200c12aa9db817d4d9f0a
>[heketi] INFO 2018/06/08 08:54:22 Allocating brick set #0
>[negroni] Completed 202 Accepted in 5m2.127609089s
>[asynchttp] INFO 2018/06/08 08:54:22 asynchttp.go:288: Started job addeaaa11160c5240019710f9906a045
>[heketi] INFO 2018/06/08 08:54:22 Started async operation: Create Volume
>[heketi] INFO 2018/06/08 08:54:22 Creating brick b35881161d7bf25bfa4af1bea4746d58
>[heketi] INFO 2018/06/08 08:54:22 Creating brick 4fd98fc9e1c8a606b4b5c70bb555597e
>[heketi] INFO 2018/06/08 08:54:22 Creating brick a84d4290cbebd984dd11b88eb806d93b
>[heketi] INFO 2018/06/08 08:54:22 Allocating brick set #0
>[kubeexec] DEBUG 2018/06/08 08:54:22 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir -p /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_cd0ffc6bd88bdb1d969dceaafda517d8
>Result: 
>[negroni] Completed 202 Accepted in 5m2.152129648s
>[asynchttp] INFO 2018/06/08 08:54:22 asynchttp.go:288: Started job e437810dbd1e27db60605a0a68708d27
>[heketi] INFO 2018/06/08 08:54:22 Started async operation: Create Volume
>[heketi] INFO 2018/06/08 08:54:22 Creating brick 2bba219914cb29c5994021b1731b7763
>[heketi] INFO 2018/06/08 08:54:22 Creating brick bf4ebc8a0c81383f3dc357f04e4903c1
>[heketi] INFO 2018/06/08 08:54:22 Creating brick 8546c963be0cfb030d57435e94a50d61
>[heketi] INFO 2018/06/08 08:54:23 Allocating brick set #0
>[negroni] Completed 202 Accepted in 5m2.182011023s
>[asynchttp] INFO 2018/06/08 08:54:23 asynchttp.go:288: Started job 3addf5576b966f87a33743f58885b381
>[heketi] INFO 2018/06/08 08:54:23 Started async operation: Create Volume
>[heketi] INFO 2018/06/08 08:54:23 Creating brick 6167bec47d9258cf5b2a5e3a48c0d391
>[heketi] INFO 2018/06/08 08:54:23 Creating brick 6319c734fbf95bbbfb1156b24dcc43ea
>[heketi] INFO 2018/06/08 08:54:23 Creating brick b2827681c0e1655380b4d5d53226bf7e
>[heketi] INFO 2018/06/08 08:54:23 Allocating brick set #0
>[negroni] Completed 202 Accepted in 5m2.21540069s
>[asynchttp] INFO 2018/06/08 08:54:23 asynchttp.go:288: Started job 9a79c05867577fc3b0a49e88846f7d5d
>[heketi] INFO 2018/06/08 08:54:23 Started async operation: Create Volume
>[heketi] INFO 2018/06/08 08:54:23 Creating brick 9963c5d95158f58c8fc2f28d81d925ea
>[heketi] INFO 2018/06/08 08:54:23 Creating brick 322eeb9ae02c4f9e338567566394d2c7
>[heketi] INFO 2018/06/08 08:54:23 Creating brick a1150dfd2cad12bad69ef9cc830da83d
>[heketi] INFO 2018/06/08 08:54:23 Allocating brick set #0
>[negroni] Completed 202 Accepted in 5m2.247836708s
>[asynchttp] INFO 2018/06/08 08:54:23 asynchttp.go:288: Started job 07364f7e79063d4523983c360e41ce2a
>[heketi] INFO 2018/06/08 08:54:23 Started async operation: Create Volume
>[heketi] INFO 2018/06/08 08:54:23 Creating brick 211fca2005300a58cf3b464b89e9deea
>[heketi] INFO 2018/06/08 08:54:23 Creating brick 4d4160003ee228514a59c2c61932ceed
>[heketi] INFO 2018/06/08 08:54:23 Creating brick 3ca7a79bfe1ee21131ef20732eac93d7
>[heketi] INFO 2018/06/08 08:54:23 Allocating brick set #0
>[negroni] Completed 202 Accepted in 5m2.283145242s
>[asynchttp] INFO 2018/06/08 08:54:23 asynchttp.go:288: Started job be6e7eb7be5d4288a84770ab21a780d4
>[heketi] INFO 2018/06/08 08:54:23 Started async operation: Create Volume
>[heketi] ERROR 2018/06/08 08:54:23 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:399: Pending brick 214eb9006f9103530a1d0310d5f5dcfc can not be deleted
>[heketi] INFO 2018/06/08 08:54:23 Creating brick 30eb22cad2c3536d381b8d71943cffd2
>[heketi] INFO 2018/06/08 08:54:23 Creating brick df153caad54b6511504d5af0cdc4e2b5
>[heketi] INFO 2018/06/08 08:54:23 Creating brick 54258f15bd3df0dfb8421196bdea9774
>[negroni] Completed 500 Internal Server Error in 4m47.301233838s
>[heketi] ERROR 2018/06/08 08:54:23 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:1167: Delete Volume Build Failed: The target exists, contains other items, or is in use.
>[heketi] INFO 2018/06/08 08:54:23 Allocating brick set #0
>[negroni] Completed 202 Accepted in 4m29.35565232s
>[asynchttp] INFO 2018/06/08 08:54:23 asynchttp.go:288: Started job cba4a2a4640269f19e7dad88f97689c6
>[heketi] INFO 2018/06/08 08:54:23 Started async operation: Create Volume
>[heketi] INFO 2018/06/08 08:54:23 Creating brick 02dbc9628a1cd8a5f20d9fea7a615dcc
>[heketi] INFO 2018/06/08 08:54:23 Creating brick a69c00298f148d2208ea7eab0903a3d2
>[heketi] INFO 2018/06/08 08:54:23 Creating brick e1709c1db8499b885817e5f4682c56ca
>[heketi] INFO 2018/06/08 08:54:23 Allocating brick set #0
>[negroni] Completed 202 Accepted in 4m29.176267251s
>[asynchttp] INFO 2018/06/08 08:54:23 asynchttp.go:288: Started job feaab31cecfc86532b5845f95cf63913
>[heketi] INFO 2018/06/08 08:54:23 Started async operation: Create Volume
>[heketi] INFO 2018/06/08 08:54:23 Creating brick b4e462e2cc3dbccfd86e44f907dc7f00
>[heketi] INFO 2018/06/08 08:54:23 Creating brick 2a97d96e94e0ff7c49d2b6d81dfbd8fd
>[heketi] INFO 2018/06/08 08:54:23 Creating brick dde70bf2e86fac7df0e0427af9bf5db3
>[heketi] INFO 2018/06/08 08:54:23 Allocating brick set #0
>[negroni] Completed 202 Accepted in 4m28.969538535s
>[asynchttp] INFO 2018/06/08 08:54:23 asynchttp.go:288: Started job 85bc22f147332f528f993604639f4b29
>[heketi] INFO 2018/06/08 08:54:23 Started async operation: Create Volume
>[heketi] INFO 2018/06/08 08:54:23 Creating brick d1e261647e36a4dd323508df5b3decff
>[heketi] INFO 2018/06/08 08:54:23 Creating brick da658ca7b4961a716b886b57a67ff000
>[heketi] INFO 2018/06/08 08:54:23 Creating brick 906b174d0f419aa3d1a9affe4675674a
>[heketi] ERROR 2018/06/08 08:54:23 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:1185: Delete Volume Failed: Unable to delete volume vol_966a92ed3e4374a5e634f7b133c49e52: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_966a92ed3e4374a5e634f7b133c49e52: failed: Another transaction is in progress for vol_966a92ed3e4374a5e634f7b133c49e52. Please try again after sometime.
>[asynchttp] INFO 2018/06/08 08:54:23 asynchttp.go:292: Completed job 9abffd27d4b48c8f2f0e699632a69408 in 8m17.439460258s
>[kubeexec] DEBUG 2018/06/08 08:54:23 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_9394bc70699b006c5460c9f654cf345f/tp_cd0ffc6bd88bdb1d969dceaafda517d8 --virtualsize 2097152K --name brick_cd0ffc6bd88bdb1d969dceaafda517d8
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_cd0ffc6bd88bdb1d969dceaafda517d8" created.
>[heketi] INFO 2018/06/08 08:54:23 Allocating brick set #0
>[negroni] Completed 202 Accepted in 4m28.766518796s
>[asynchttp] INFO 2018/06/08 08:54:23 asynchttp.go:288: Started job b6fa7484737d1ad70ee715a59b9488ee
>[heketi] INFO 2018/06/08 08:54:23 Started async operation: Create Volume
>[heketi] INFO 2018/06/08 08:54:23 Creating brick 0a39774f01ac7959661ff5afff2595ab
>[heketi] INFO 2018/06/08 08:54:23 Creating brick 592a78c302b01c3ee9538269481405e7
>[heketi] INFO 2018/06/08 08:54:23 Creating brick 11a59fea08d9b732822c2669ddaf54fa
>[heketi] INFO 2018/06/08 08:54:23 Allocating brick set #0
>[negroni] Completed 202 Accepted in 4m28.613555187s
>[asynchttp] INFO 2018/06/08 08:54:23 asynchttp.go:288: Started job 2d92a9e5a1719ab40f22667eb62b0d09
>[heketi] INFO 2018/06/08 08:54:23 Started async operation: Create Volume
>[heketi] INFO 2018/06/08 08:54:23 Creating brick b3918aae89c9aab7c5cfa3496f95936a
>[heketi] INFO 2018/06/08 08:54:23 Creating brick dba8827a8c1b642d1b34bc5cf35aa4b4
>[heketi] INFO 2018/06/08 08:54:23 Creating brick c8dd6b39719a6bc75ea331fae5a92396
>[heketi] INFO 2018/06/08 08:54:23 Allocating brick set #0
>[negroni] Completed 202 Accepted in 4m28.4586346s
>[asynchttp] INFO 2018/06/08 08:54:23 asynchttp.go:288: Started job 5a4d86d3903a860d822a06fd74343b52
>[heketi] INFO 2018/06/08 08:54:23 Started async operation: Create Volume
>[heketi] INFO 2018/06/08 08:54:23 Creating brick 9f2b3abdf7a755d6b95c412c440a955f
>[heketi] INFO 2018/06/08 08:54:23 Creating brick 01b4fda3de4a6a9df5de0de9826ad1aa
>[heketi] INFO 2018/06/08 08:54:23 Creating brick e54cabdc56fef4e5b4a11c7a72eaff3d
>[heketi] INFO 2018/06/08 08:54:23 Allocating brick set #0
>[negroni] Completed 202 Accepted in 4m28.278139136s
>[asynchttp] INFO 2018/06/08 08:54:23 asynchttp.go:288: Started job edf46a25868eb4962cc099193bc77737
>[heketi] INFO 2018/06/08 08:54:23 Started async operation: Create Volume
>[kubeexec] DEBUG 2018/06/08 08:54:23 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_cd0ffc6bd88bdb1d969dceaafda517d8
>Result: meta-data=/dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_cd0ffc6bd88bdb1d969dceaafda517d8 isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[heketi] INFO 2018/06/08 08:54:23 Creating brick e2e58ed3bab4af0c07b035a7306264ab
>[heketi] INFO 2018/06/08 08:54:23 Creating brick f399afe93703292ed5ac22602d678d6f
>[heketi] INFO 2018/06/08 08:54:23 Creating brick 68704fc0eb854fe2a0157b0261302792
>[heketi] INFO 2018/06/08 08:54:23 Allocating brick set #0
>[negroni] Completed 202 Accepted in 4m26.777086971s
>[asynchttp] INFO 2018/06/08 08:54:23 asynchttp.go:288: Started job 01bc6bf544472a829302d04d4df8b0d1
>[heketi] INFO 2018/06/08 08:54:23 Started async operation: Create Volume
>[heketi] ERROR 2018/06/08 08:54:23 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:399: Pending brick 214eb9006f9103530a1d0310d5f5dcfc can not be deleted
>[heketi] ERROR 2018/06/08 08:54:23 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:1167: Delete Volume Build Failed: The target exists, contains other items, or is in use.
>[heketi] INFO 2018/06/08 08:54:23 Creating brick d105708cfed6c89ba77c7f9738020bf4
>[heketi] INFO 2018/06/08 08:54:23 Creating brick ad542929ce4fd8719fbe5fc44df98dbd
>[heketi] INFO 2018/06/08 08:54:23 Creating brick 4f44e68452214ab813c4616ad12e2ee2
>[negroni] Completed 500 Internal Server Error in 4m2.718155106s
>[heketi] INFO 2018/06/08 08:54:23 Allocating brick set #0
>[heketi] INFO 2018/06/08 08:54:23 Allocating brick set #0
>[heketi] INFO 2018/06/08 08:54:23 Allocating brick set #1
>[heketi] INFO 2018/06/08 08:54:23 Allocating brick set #0
>[heketi] INFO 2018/06/08 08:54:23 Allocating brick set #1
>[heketi] INFO 2018/06/08 08:54:23 Allocating brick set #2
>[heketi] INFO 2018/06/08 08:54:23 Allocating brick set #3
>[heketi] INFO 2018/06/08 08:54:23 Allocating brick set #0
>[heketi] INFO 2018/06/08 08:54:23 Allocating brick set #1
>[heketi] INFO 2018/06/08 08:54:23 Allocating brick set #2
>[heketi] INFO 2018/06/08 08:54:23 Allocating brick set #3
>[heketi] ERROR 2018/06/08 08:54:23 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:1167: Create Volume Build Failed: No space
>[heketi] INFO 2018/06/08 08:54:23 Allocating brick set #4
>[heketi] INFO 2018/06/08 08:54:23 Allocating brick set #5
>[heketi] INFO 2018/06/08 08:54:23 Allocating brick set #6
>[negroni] Completed 500 Internal Server Error in 3m55.963669042s
>[heketi] INFO 2018/06/08 08:54:23 Allocating brick set #0
>[heketi] INFO 2018/06/08 08:54:23 Allocating brick set #0
>[heketi] INFO 2018/06/08 08:54:23 Allocating brick set #1
>[heketi] INFO 2018/06/08 08:54:23 Allocating brick set #0
>[heketi] INFO 2018/06/08 08:54:23 Allocating brick set #1
>[heketi] INFO 2018/06/08 08:54:23 Allocating brick set #2
>[heketi] INFO 2018/06/08 08:54:23 Allocating brick set #3
>[heketi] INFO 2018/06/08 08:54:23 Allocating brick set #0
>[heketi] INFO 2018/06/08 08:54:23 Allocating brick set #1
>[heketi] INFO 2018/06/08 08:54:23 Allocating brick set #2
>[heketi] INFO 2018/06/08 08:54:23 Allocating brick set #3
>[heketi] INFO 2018/06/08 08:54:23 Allocating brick set #4
>[heketi] INFO 2018/06/08 08:54:23 Allocating brick set #5
>[heketi] INFO 2018/06/08 08:54:23 Allocating brick set #6
>[heketi] ERROR 2018/06/08 08:54:23 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:1167: Create Volume Build Failed: No space
>[negroni] Completed 500 Internal Server Error in 3m25.837382141s
>[heketi] ERROR 2018/06/08 08:54:23 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:399: Pending brick 214eb9006f9103530a1d0310d5f5dcfc can not be deleted
>[heketi] ERROR 2018/06/08 08:54:23 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:1167: Delete Volume Build Failed: The target exists, contains other items, or is in use.
>[negroni] Completed 500 Internal Server Error in 3m17.769667715s
>[kubeexec] DEBUG 2018/06/08 08:54:23 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: awk "BEGIN {print \"/dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_cd0ffc6bd88bdb1d969dceaafda517d8 /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_cd0ffc6bd88bdb1d969dceaafda517d8 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[heketi] INFO 2018/06/08 08:54:23 Allocating brick set #0
>[negroni] Completed 202 Accepted in 2m54.9643561s
>[asynchttp] INFO 2018/06/08 08:54:23 asynchttp.go:288: Started job a85d28ab7c8834b5c41247d658312b8f
>[heketi] INFO 2018/06/08 08:54:23 Started async operation: Create Volume
>[heketi] INFO 2018/06/08 08:54:23 Creating brick fcac0ae8bd9895d4780d78b18d3d6c38
>[heketi] INFO 2018/06/08 08:54:23 Creating brick c36c6529173a51e7b9ae7a98545d98a2
>[heketi] INFO 2018/06/08 08:54:23 Creating brick 1039ed81d7d98ea183e9c7d8d00c1b6d
>[heketi] INFO 2018/06/08 08:54:23 Allocating brick set #0
>[negroni] Completed 202 Accepted in 2m24.28656674s
>[asynchttp] INFO 2018/06/08 08:54:23 asynchttp.go:288: Started job 1b65bda5683c59bd435d17b7a832a860
>[heketi] INFO 2018/06/08 08:54:23 Started async operation: Create Volume
>[heketi] ERROR 2018/06/08 08:54:23 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:399: Pending brick 214eb9006f9103530a1d0310d5f5dcfc can not be deleted
>[heketi] INFO 2018/06/08 08:54:23 Creating brick e1c7bf2add37671886c54c55f98f9fb7
>[heketi] ERROR 2018/06/08 08:54:23 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:1167: Delete Volume Build Failed: The target exists, contains other items, or is in use.
>[negroni] Completed 500 Internal Server Error in 2m17.823202408s
>[heketi] INFO 2018/06/08 08:54:23 Creating brick f26cbb3df4880b3fe991ee4e44697c2c
>[heketi] INFO 2018/06/08 08:54:23 Creating brick 1f46651d33f6ef49271d6d5382a7bc9c
>[heketi] WARNING 2018/06/08 08:54:23 Unable to delete cluster [0a73c60efdd4673113b668afea101e6d] because it contains volumes and/or nodes
>[negroni] Completed 409 Conflict in 1m53.881721054s
>[heketi] INFO 2018/06/08 08:54:23 Allocating brick set #0
>[negroni] Completed 202 Accepted in 1m22.896263777s
>[asynchttp] INFO 2018/06/08 08:54:23 asynchttp.go:288: Started job fe707ace9f825f3ad19b3b8f28410ae4
>[heketi] INFO 2018/06/08 08:54:23 Started async operation: Create Volume
>[heketi] ERROR 2018/06/08 08:54:23 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:399: Pending brick 214eb9006f9103530a1d0310d5f5dcfc can not be deleted
>[heketi] ERROR 2018/06/08 08:54:23 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:1167: Delete Volume Build Failed: The target exists, contains other items, or is in use.
>[negroni] Completed 500 Internal Server Error in 1m2.857919329s
>[heketi] INFO 2018/06/08 08:54:23 Creating brick e631f0bf543f6c06867077cd16aad9e2
>[heketi] INFO 2018/06/08 08:54:23 Creating brick 04daa5c9d25bc1a3074533508d73b587
>[heketi] INFO 2018/06/08 08:54:23 Creating brick 61815093958a51df17cf62e0a12a5451
>[heketi] INFO 2018/06/08 08:54:23 Create Volume succeeded
>[asynchttp] INFO 2018/06/08 08:54:23 asynchttp.go:292: Completed job d3134ebfac4ef9bd8dbf824bd7bf873a in 2m29.176160341s
>[heketi] INFO 2018/06/08 08:54:23 Create Volume succeeded
>[asynchttp] INFO 2018/06/08 08:54:23 asynchttp.go:292: Completed job 53ae2da32657329fa5a30bfda1414335 in 28.933993184s
>[negroni] Started GET /queue/548c6fbe203bb93f9a4ecb56f976926d
>[negroni] Completed 500 Internal Server Error in 105.1Âµs
>[negroni] Completed 202 Accepted in 2.892508234s
>[asynchttp] INFO 2018/06/08 08:54:23 asynchttp.go:288: Started job 35a61a66b97e9b1b5514e032177179ad
>[heketi] INFO 2018/06/08 08:54:23 Started async operation: Delete Volume
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 85.236Âµs
>[kubeexec] DEBUG 2018/06/08 08:54:23 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_cd0ffc6bd88bdb1d969dceaafda517d8 /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_cd0ffc6bd88bdb1d969dceaafda517d8
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:23 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_cd0ffc6bd88bdb1d969dceaafda517d8/brick
>Result: 
>[negroni] Started GET /queue/9abffd27d4b48c8f2f0e699632a69408
>[negroni] Completed 500 Internal Server Error in 92.158Âµs
>[kubeexec] DEBUG 2018/06/08 08:54:23 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: chown :2001 /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_cd0ffc6bd88bdb1d969dceaafda517d8/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:24 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: chmod 2775 /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_cd0ffc6bd88bdb1d969dceaafda517d8/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:24 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mount | grep -w /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_4f4d753d298c99eac492c32006c74484 | cut -d" " -f1
>Result: /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_4f4d753d298c99eac492c32006c74484
>[kubeexec] DEBUG 2018/06/08 08:54:24 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mount | grep -w /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_2ed96343627983cf9667b7cee4052d17 | cut -d" " -f1
>Result: /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_2ed96343627983cf9667b7cee4052d17
>[kubeexec] DEBUG 2018/06/08 08:54:24 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir -p /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_e3ebda9db9a085114fe9732ab2df4869
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:24 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_9394bc70699b006c5460c9f654cf345f/tp_e3ebda9db9a085114fe9732ab2df4869 --virtualsize 2097152K --name brick_e3ebda9db9a085114fe9732ab2df4869
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_e3ebda9db9a085114fe9732ab2df4869" created.
>[kubeexec] DEBUG 2018/06/08 08:54:24 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_e3ebda9db9a085114fe9732ab2df4869
>Result: meta-data=/dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_e3ebda9db9a085114fe9732ab2df4869 isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 110.439Âµs
>[kubeexec] DEBUG 2018/06/08 08:54:24 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: awk "BEGIN {print \"/dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_e3ebda9db9a085114fe9732ab2df4869 /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_e3ebda9db9a085114fe9732ab2df4869 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:24 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: gluster --mode=script volume stop vol_ad1e5849e9566f1bcaa09cfb9c0b96ef force
>Result: volume stop: vol_ad1e5849e9566f1bcaa09cfb9c0b96ef: success
>[kubeexec] DEBUG 2018/06/08 08:54:24 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script volume create vol_2bf097c60bd8b38bfcb4327727ca5681 replica 3 10.70.46.122:/var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_35ab4a65e06fcf26cf320c522f9afc92/brick 10.70.47.76:/var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_e0e48c633e08f573cc3b70b99f47faea/brick 10.70.46.187:/var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_6f7534e6df0ad44d669269ef560a18ed/brick
>Result: volume create: vol_2bf097c60bd8b38bfcb4327727ca5681: success: please start the volume to access data
>[kubeexec] DEBUG 2018/06/08 08:54:24 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_e3ebda9db9a085114fe9732ab2df4869 /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_e3ebda9db9a085114fe9732ab2df4869
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:24 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir -p /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_6bfb8b3bea31f51d5be26acc7e81ce9a
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:25 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_e3ebda9db9a085114fe9732ab2df4869/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:25 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: chown :2002 /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_e3ebda9db9a085114fe9732ab2df4869/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:25 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: chmod 2775 /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_e3ebda9db9a085114fe9732ab2df4869/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:25 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_b861af9cc67f5bf8d5a8f9e38f56b14e/tp_6bfb8b3bea31f51d5be26acc7e81ce9a --virtualsize 2097152K --name brick_6bfb8b3bea31f51d5be26acc7e81ce9a
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_6bfb8b3bea31f51d5be26acc7e81ce9a" created.
>[kubeexec] DEBUG 2018/06/08 08:54:25 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir -p /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_abd8495bfd886d541a449967f7bf70c0
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:25 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_6bfb8b3bea31f51d5be26acc7e81ce9a
>Result: meta-data=/dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_6bfb8b3bea31f51d5be26acc7e81ce9a isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:54:25 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: awk "BEGIN {print \"/dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_6bfb8b3bea31f51d5be26acc7e81ce9a /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_6bfb8b3bea31f51d5be26acc7e81ce9a xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:25 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_9394bc70699b006c5460c9f654cf345f/tp_abd8495bfd886d541a449967f7bf70c0 --virtualsize 2097152K --name brick_abd8495bfd886d541a449967f7bf70c0
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_abd8495bfd886d541a449967f7bf70c0" created.
>[kubeexec] DEBUG 2018/06/08 08:54:25 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_6bfb8b3bea31f51d5be26acc7e81ce9a /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_6bfb8b3bea31f51d5be26acc7e81ce9a
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:25 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_6bfb8b3bea31f51d5be26acc7e81ce9a/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:25 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_abd8495bfd886d541a449967f7bf70c0
>Result: meta-data=/dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_abd8495bfd886d541a449967f7bf70c0 isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 236.457Âµs
>[kubeexec] DEBUG 2018/06/08 08:54:25 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: chown :2007 /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_6bfb8b3bea31f51d5be26acc7e81ce9a/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:25 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: awk "BEGIN {print \"/dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_abd8495bfd886d541a449967f7bf70c0 /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_abd8495bfd886d541a449967f7bf70c0 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:25 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: chmod 2775 /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_6bfb8b3bea31f51d5be26acc7e81ce9a/brick
>Result: 
>[cmdexec] INFO 2018/06/08 08:54:25 Creating volume vol_07ef5105131fa51c35a9007ee213ea7a replica 3
>[kubeexec] DEBUG 2018/06/08 08:54:25 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_abd8495bfd886d541a449967f7bf70c0 /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_abd8495bfd886d541a449967f7bf70c0
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:25 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir -p /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_cb9dab914105ca3d691abed1d53d7df9
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:26 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_abd8495bfd886d541a449967f7bf70c0/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:26 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: chown :2003 /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_abd8495bfd886d541a449967f7bf70c0/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:26 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: chmod 2775 /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_abd8495bfd886d541a449967f7bf70c0/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:26 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_b861af9cc67f5bf8d5a8f9e38f56b14e/tp_cb9dab914105ca3d691abed1d53d7df9 --virtualsize 2097152K --name brick_cb9dab914105ca3d691abed1d53d7df9
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_cb9dab914105ca3d691abed1d53d7df9" created.
>[kubeexec] DEBUG 2018/06/08 08:54:26 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir -p /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_24c8f78999c94f4eacb087ac1fc563d1
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:26 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_cb9dab914105ca3d691abed1d53d7df9
>Result: meta-data=/dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_cb9dab914105ca3d691abed1d53d7df9 isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:54:26 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: awk "BEGIN {print \"/dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_cb9dab914105ca3d691abed1d53d7df9 /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_cb9dab914105ca3d691abed1d53d7df9 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:26 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_cb9dab914105ca3d691abed1d53d7df9 /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_cb9dab914105ca3d691abed1d53d7df9
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:26 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_96f1667f2f1ced2c5ef94772922be93b/tp_24c8f78999c94f4eacb087ac1fc563d1 --virtualsize 2097152K --name brick_24c8f78999c94f4eacb087ac1fc563d1
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_24c8f78999c94f4eacb087ac1fc563d1" created.
>[kubeexec] DEBUG 2018/06/08 08:54:26 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_cb9dab914105ca3d691abed1d53d7df9/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:26 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_24c8f78999c94f4eacb087ac1fc563d1
>Result: meta-data=/dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_24c8f78999c94f4eacb087ac1fc563d1 isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 107.453Âµs
>[kubeexec] DEBUG 2018/06/08 08:54:26 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: chown :2001 /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_cb9dab914105ca3d691abed1d53d7df9/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:26 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: awk "BEGIN {print \"/dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_24c8f78999c94f4eacb087ac1fc563d1 /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_24c8f78999c94f4eacb087ac1fc563d1 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:26 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: chmod 2775 /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_cb9dab914105ca3d691abed1d53d7df9/brick
>Result: 
>[cmdexec] INFO 2018/06/08 08:54:26 Creating volume vol_a6be87754541710c38b420381c76fb8c replica 3
>[kubeexec] DEBUG 2018/06/08 08:54:26 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mount | grep -w /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_f05028077b974ebc1f6621aee2184169 | cut -d" " -f1
>Result: /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_f05028077b974ebc1f6621aee2184169
>[kubeexec] DEBUG 2018/06/08 08:54:26 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_24c8f78999c94f4eacb087ac1fc563d1 /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_24c8f78999c94f4eacb087ac1fc563d1
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:27 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mount | grep -w /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_1bb49691eccf328025855a91ee8cbc66 | cut -d" " -f1
>Result: /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_1bb49691eccf328025855a91ee8cbc66
>[kubeexec] DEBUG 2018/06/08 08:54:27 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_24c8f78999c94f4eacb087ac1fc563d1/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:27 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: chown :2004 /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_24c8f78999c94f4eacb087ac1fc563d1/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:27 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: chmod 2775 /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_24c8f78999c94f4eacb087ac1fc563d1/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:27 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir -p /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_9be3d8f4748b4c09541f8a9de90c4c34
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:27 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_96f1667f2f1ced2c5ef94772922be93b/tp_9be3d8f4748b4c09541f8a9de90c4c34 --virtualsize 2097152K --name brick_9be3d8f4748b4c09541f8a9de90c4c34
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_9be3d8f4748b4c09541f8a9de90c4c34" created.
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 122.633Âµs
>[kubeexec] DEBUG 2018/06/08 08:54:27 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_9be3d8f4748b4c09541f8a9de90c4c34
>Result: meta-data=/dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_9be3d8f4748b4c09541f8a9de90c4c34 isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:54:27 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: awk "BEGIN {print \"/dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_9be3d8f4748b4c09541f8a9de90c4c34 /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_9be3d8f4748b4c09541f8a9de90c4c34 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:28 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_9be3d8f4748b4c09541f8a9de90c4c34 /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_9be3d8f4748b4c09541f8a9de90c4c34
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:28 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_9be3d8f4748b4c09541f8a9de90c4c34/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:28 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: chown :2005 /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_9be3d8f4748b4c09541f8a9de90c4c34/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:28 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: chmod 2775 /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_9be3d8f4748b4c09541f8a9de90c4c34/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:28 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir -p /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_ac27bcfa769ceba759b678d9e6b751cb
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:28 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_9394bc70699b006c5460c9f654cf345f/tp_ac27bcfa769ceba759b678d9e6b751cb --virtualsize 2097152K --name brick_ac27bcfa769ceba759b678d9e6b751cb
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_ac27bcfa769ceba759b678d9e6b751cb" created.
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 140.129Âµs
>[kubeexec] DEBUG 2018/06/08 08:54:28 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_ac27bcfa769ceba759b678d9e6b751cb
>Result: meta-data=/dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_ac27bcfa769ceba759b678d9e6b751cb isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:54:28 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: awk "BEGIN {print \"/dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_ac27bcfa769ceba759b678d9e6b751cb /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_ac27bcfa769ceba759b678d9e6b751cb xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:28 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_ac27bcfa769ceba759b678d9e6b751cb /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_ac27bcfa769ceba759b678d9e6b751cb
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:29 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_ac27bcfa769ceba759b678d9e6b751cb/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:29 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script volume start vol_2bf097c60bd8b38bfcb4327727ca5681
>Result: volume start: vol_2bf097c60bd8b38bfcb4327727ca5681: success
>[kubeexec] ERROR 2018/06/08 08:54:29 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume stop vol_b99532640a5201d243193159ee762ae4 force] on glusterfs-storage-vsh2m: Err[command terminated with exit code 1]: Stdout []: Stderr [volume stop: vol_b99532640a5201d243193159ee762ae4: failed: Volume vol_b99532640a5201d243193159ee762ae4 does not exist
>]
>[cmdexec] ERROR 2018/06/08 08:54:29 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:144: Unable to stop volume vol_b99532640a5201d243193159ee762ae4: Unable to execute command on glusterfs-storage-vsh2m: volume stop: vol_b99532640a5201d243193159ee762ae4: failed: Volume vol_b99532640a5201d243193159ee762ae4 does not exist
>[kubeexec] DEBUG 2018/06/08 08:54:29 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: chown :2007 /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_ac27bcfa769ceba759b678d9e6b751cb/brick
>Result: 
>[heketi] INFO 2018/06/08 08:54:29 Create Volume succeeded
>[asynchttp] INFO 2018/06/08 08:54:29 asynchttp.go:292: Completed job c2300af895526bf82435fd82aaea0ce8 in 30.638297718s
>[kubeexec] DEBUG 2018/06/08 08:54:29 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: chmod 2775 /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_ac27bcfa769ceba759b678d9e6b751cb/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:29 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir -p /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_c4423da2a93ab3c5917479262cf1d93a
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:29 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_cc2483d7b49bc029b5200a024cac7535
>
>Result:   vg_3a4297677881963e3f80124971d50eea/tp_cc2483d7b49bc029b5200a024cac7535
>[kubeexec] DEBUG 2018/06/08 08:54:29 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir -p /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_d8a382627ef730cbe0664d9439fd6a59
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:29 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvs --noheadings --options=thin_count   vg_66a9af9f6bac95f9d8d556a2f14c29d3/tp_214eb9006f9103530a1d0310d5f5dcfc
>
>Result:        0
>[kubeexec] DEBUG 2018/06/08 08:54:29 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_d389f0278a774bd7443a09af960961d8/tp_c4423da2a93ab3c5917479262cf1d93a --virtualsize 2097152K --name brick_c4423da2a93ab3c5917479262cf1d93a
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_c4423da2a93ab3c5917479262cf1d93a" created.
>[kubeexec] DEBUG 2018/06/08 08:54:29 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_96f1667f2f1ced2c5ef94772922be93b/tp_d8a382627ef730cbe0664d9439fd6a59 --virtualsize 2097152K --name brick_d8a382627ef730cbe0664d9439fd6a59
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_d8a382627ef730cbe0664d9439fd6a59" created.
>[kubeexec] DEBUG 2018/06/08 08:54:29 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_c4423da2a93ab3c5917479262cf1d93a
>Result: meta-data=/dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_c4423da2a93ab3c5917479262cf1d93a isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 103.993Âµs
>[kubeexec] DEBUG 2018/06/08 08:54:29 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: awk "BEGIN {print \"/dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_c4423da2a93ab3c5917479262cf1d93a /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_c4423da2a93ab3c5917479262cf1d93a xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:29 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_d8a382627ef730cbe0664d9439fd6a59
>Result: meta-data=/dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_d8a382627ef730cbe0664d9439fd6a59 isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:54:29 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: awk "BEGIN {print \"/dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_d8a382627ef730cbe0664d9439fd6a59 /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_d8a382627ef730cbe0664d9439fd6a59 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:29 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_c4423da2a93ab3c5917479262cf1d93a /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_c4423da2a93ab3c5917479262cf1d93a
>Result: 
>[kubeexec] ERROR 2018/06/08 08:54:29 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume stop vol_15e0122e942fc41f80666a3714670682 force] on glusterfs-storage-pg4xc: Err[command terminated with exit code 1]: Stdout []: Stderr [volume stop: vol_15e0122e942fc41f80666a3714670682: failed: Volume vol_15e0122e942fc41f80666a3714670682 does not exist
>]
>[cmdexec] ERROR 2018/06/08 08:54:29 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:144: Unable to stop volume vol_15e0122e942fc41f80666a3714670682: Unable to execute command on glusterfs-storage-pg4xc: volume stop: vol_15e0122e942fc41f80666a3714670682: failed: Volume vol_15e0122e942fc41f80666a3714670682 does not exist
>[kubeexec] DEBUG 2018/06/08 08:54:29 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_c4423da2a93ab3c5917479262cf1d93a/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:29 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_d8a382627ef730cbe0664d9439fd6a59 /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_d8a382627ef730cbe0664d9439fd6a59
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:30 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: chown :2002 /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_c4423da2a93ab3c5917479262cf1d93a/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:30 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_d8a382627ef730cbe0664d9439fd6a59/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:30 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: chmod 2775 /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_c4423da2a93ab3c5917479262cf1d93a/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:30 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: chown :2006 /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_d8a382627ef730cbe0664d9439fd6a59/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:30 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir -p /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_91d54591c1b07bc3a93d925cfbbd0f23
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:30 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: chmod 2775 /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_d8a382627ef730cbe0664d9439fd6a59/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:30 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_66a9af9f6bac95f9d8d556a2f14c29d3/tp_91d54591c1b07bc3a93d925cfbbd0f23 --virtualsize 2097152K --name brick_91d54591c1b07bc3a93d925cfbbd0f23
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_91d54591c1b07bc3a93d925cfbbd0f23" created.
>[kubeexec] DEBUG 2018/06/08 08:54:30 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_91d54591c1b07bc3a93d925cfbbd0f23
>Result: meta-data=/dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_91d54591c1b07bc3a93d925cfbbd0f23 isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:54:30 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: awk "BEGIN {print \"/dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_91d54591c1b07bc3a93d925cfbbd0f23 /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_91d54591c1b07bc3a93d925cfbbd0f23 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 100.416Âµs
>[kubeexec] DEBUG 2018/06/08 08:54:30 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir -p /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_aec48b6cbce6be7256a2b86efac6aef6
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:30 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_91d54591c1b07bc3a93d925cfbbd0f23 /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_91d54591c1b07bc3a93d925cfbbd0f23
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:30 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_91d54591c1b07bc3a93d925cfbbd0f23/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:30 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir -p /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_8d39164e3ebba9d10c8b6ed33dafabc7
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:31 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: chown :2002 /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_91d54591c1b07bc3a93d925cfbbd0f23/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:31 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_d389f0278a774bd7443a09af960961d8/tp_aec48b6cbce6be7256a2b86efac6aef6 --virtualsize 2097152K --name brick_aec48b6cbce6be7256a2b86efac6aef6
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_aec48b6cbce6be7256a2b86efac6aef6" created.
>[kubeexec] DEBUG 2018/06/08 08:54:31 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: chmod 2775 /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_91d54591c1b07bc3a93d925cfbbd0f23/brick
>Result: 
>[cmdexec] INFO 2018/06/08 08:54:31 Creating volume vol_b8f78f128f20b61ef032ce9ee5b6481c replica 3
>[kubeexec] DEBUG 2018/06/08 08:54:31 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_aec48b6cbce6be7256a2b86efac6aef6
>Result: meta-data=/dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_aec48b6cbce6be7256a2b86efac6aef6 isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:54:31 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: awk "BEGIN {print \"/dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_aec48b6cbce6be7256a2b86efac6aef6 /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_aec48b6cbce6be7256a2b86efac6aef6 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:31 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_96f1667f2f1ced2c5ef94772922be93b/tp_8d39164e3ebba9d10c8b6ed33dafabc7 --virtualsize 2097152K --name brick_8d39164e3ebba9d10c8b6ed33dafabc7
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_8d39164e3ebba9d10c8b6ed33dafabc7" created.
>[kubeexec] DEBUG 2018/06/08 08:54:31 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir -p /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_26379a9db79930ad00a37453b492ebac
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:31 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_8d39164e3ebba9d10c8b6ed33dafabc7
>Result: meta-data=/dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_8d39164e3ebba9d10c8b6ed33dafabc7 isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:54:31 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_aec48b6cbce6be7256a2b86efac6aef6 /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_aec48b6cbce6be7256a2b86efac6aef6
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:31 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: awk "BEGIN {print \"/dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_8d39164e3ebba9d10c8b6ed33dafabc7 /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_8d39164e3ebba9d10c8b6ed33dafabc7 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:31 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_aec48b6cbce6be7256a2b86efac6aef6/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:31 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: chown :2003 /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_aec48b6cbce6be7256a2b86efac6aef6/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:31 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_3a4297677881963e3f80124971d50eea/tp_26379a9db79930ad00a37453b492ebac --virtualsize 2097152K --name brick_26379a9db79930ad00a37453b492ebac
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_26379a9db79930ad00a37453b492ebac" created.
>[kubeexec] DEBUG 2018/06/08 08:54:31 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_8d39164e3ebba9d10c8b6ed33dafabc7 /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_8d39164e3ebba9d10c8b6ed33dafabc7
>Result: 
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 135.032Âµs
>[kubeexec] DEBUG 2018/06/08 08:54:31 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: chmod 2775 /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_aec48b6cbce6be7256a2b86efac6aef6/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:31 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_8d39164e3ebba9d10c8b6ed33dafabc7/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:31 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_26379a9db79930ad00a37453b492ebac
>Result: meta-data=/dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_26379a9db79930ad00a37453b492ebac isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:54:31 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir -p /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_6c8f55013bee289cd516ef97746721ff
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:31 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: chown :2001 /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_8d39164e3ebba9d10c8b6ed33dafabc7/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:31 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: awk "BEGIN {print \"/dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_26379a9db79930ad00a37453b492ebac /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_26379a9db79930ad00a37453b492ebac xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:32 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: chmod 2775 /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_8d39164e3ebba9d10c8b6ed33dafabc7/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:32 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_26379a9db79930ad00a37453b492ebac /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_26379a9db79930ad00a37453b492ebac
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:32 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_26379a9db79930ad00a37453b492ebac/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:32 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir -p /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_4fd98fc9e1c8a606b4b5c70bb555597e
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:32 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_b861af9cc67f5bf8d5a8f9e38f56b14e/tp_6c8f55013bee289cd516ef97746721ff --virtualsize 2097152K --name brick_6c8f55013bee289cd516ef97746721ff
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_6c8f55013bee289cd516ef97746721ff" created.
>[kubeexec] DEBUG 2018/06/08 08:54:32 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: chown :2003 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_26379a9db79930ad00a37453b492ebac/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:32 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_6c8f55013bee289cd516ef97746721ff
>Result: meta-data=/dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_6c8f55013bee289cd516ef97746721ff isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:54:32 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: chmod 2775 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_26379a9db79930ad00a37453b492ebac/brick
>Result: 
>[cmdexec] INFO 2018/06/08 08:54:32 Creating volume vol_dcaf25d0becadd0bb3c732c2c2ca27da replica 3
>[kubeexec] DEBUG 2018/06/08 08:54:32 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: awk "BEGIN {print \"/dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_6c8f55013bee289cd516ef97746721ff /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_6c8f55013bee289cd516ef97746721ff xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:32 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_9394bc70699b006c5460c9f654cf345f/tp_4fd98fc9e1c8a606b4b5c70bb555597e --virtualsize 2097152K --name brick_4fd98fc9e1c8a606b4b5c70bb555597e
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_4fd98fc9e1c8a606b4b5c70bb555597e" created.
>[kubeexec] DEBUG 2018/06/08 08:54:32 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir -p /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_c77746b3040810cf2ae10fc585c67fd5
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:32 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_6c8f55013bee289cd516ef97746721ff /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_6c8f55013bee289cd516ef97746721ff
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:32 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_4fd98fc9e1c8a606b4b5c70bb555597e
>Result: meta-data=/dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_4fd98fc9e1c8a606b4b5c70bb555597e isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:54:32 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_6c8f55013bee289cd516ef97746721ff/brick
>Result: 
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 136.994Âµs
>[kubeexec] DEBUG 2018/06/08 08:54:32 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: awk "BEGIN {print \"/dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_4fd98fc9e1c8a606b4b5c70bb555597e /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_4fd98fc9e1c8a606b4b5c70bb555597e xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:32 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: chown :2004 /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_6c8f55013bee289cd516ef97746721ff/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:32 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: chmod 2775 /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_6c8f55013bee289cd516ef97746721ff/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:32 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_3a4297677881963e3f80124971d50eea/tp_c77746b3040810cf2ae10fc585c67fd5 --virtualsize 2097152K --name brick_c77746b3040810cf2ae10fc585c67fd5
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_c77746b3040810cf2ae10fc585c67fd5" created.
>[kubeexec] DEBUG 2018/06/08 08:54:32 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_4fd98fc9e1c8a606b4b5c70bb555597e /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_4fd98fc9e1c8a606b4b5c70bb555597e
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:32 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir -p /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_cecca27e94262fbaae81106a8cfeb0b2
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_4fd98fc9e1c8a606b4b5c70bb555597e/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_c77746b3040810cf2ae10fc585c67fd5
>Result: meta-data=/dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_c77746b3040810cf2ae10fc585c67fd5 isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:54:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: chown :2002 /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_4fd98fc9e1c8a606b4b5c70bb555597e/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: awk "BEGIN {print \"/dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_c77746b3040810cf2ae10fc585c67fd5 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_c77746b3040810cf2ae10fc585c67fd5 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: chmod 2775 /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_4fd98fc9e1c8a606b4b5c70bb555597e/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_c77746b3040810cf2ae10fc585c67fd5 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_c77746b3040810cf2ae10fc585c67fd5
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_b861af9cc67f5bf8d5a8f9e38f56b14e/tp_cecca27e94262fbaae81106a8cfeb0b2 --virtualsize 2097152K --name brick_cecca27e94262fbaae81106a8cfeb0b2
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_cecca27e94262fbaae81106a8cfeb0b2" created.
>[kubeexec] DEBUG 2018/06/08 08:54:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_c77746b3040810cf2ae10fc585c67fd5/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir -p /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_8546c963be0cfb030d57435e94a50d61
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_cecca27e94262fbaae81106a8cfeb0b2
>Result: meta-data=/dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_cecca27e94262fbaae81106a8cfeb0b2 isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:54:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: chown :2004 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_c77746b3040810cf2ae10fc585c67fd5/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: awk "BEGIN {print \"/dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_cecca27e94262fbaae81106a8cfeb0b2 /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_cecca27e94262fbaae81106a8cfeb0b2 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: chmod 2775 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_c77746b3040810cf2ae10fc585c67fd5/brick
>Result: 
>[cmdexec] INFO 2018/06/08 08:54:33 Creating volume vol_89ebb1e7eed2ff557488996a1657e75e replica 3
>[kubeexec] DEBUG 2018/06/08 08:54:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_cecca27e94262fbaae81106a8cfeb0b2 /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_cecca27e94262fbaae81106a8cfeb0b2
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_96f1667f2f1ced2c5ef94772922be93b/tp_8546c963be0cfb030d57435e94a50d61 --virtualsize 2097152K --name brick_8546c963be0cfb030d57435e94a50d61
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_8546c963be0cfb030d57435e94a50d61" created.
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 122.117Âµs
>[kubeexec] DEBUG 2018/06/08 08:54:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_cecca27e94262fbaae81106a8cfeb0b2/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir -p /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_cdc8c0d24f68026f06c17385d2fd0029
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: chown :2005 /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_cecca27e94262fbaae81106a8cfeb0b2/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_8546c963be0cfb030d57435e94a50d61
>Result: meta-data=/dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_8546c963be0cfb030d57435e94a50d61 isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:54:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: chmod 2775 /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_cecca27e94262fbaae81106a8cfeb0b2/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: awk "BEGIN {print \"/dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_8546c963be0cfb030d57435e94a50d61 /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_8546c963be0cfb030d57435e94a50d61 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_66a9af9f6bac95f9d8d556a2f14c29d3/tp_cdc8c0d24f68026f06c17385d2fd0029 --virtualsize 2097152K --name brick_cdc8c0d24f68026f06c17385d2fd0029
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_cdc8c0d24f68026f06c17385d2fd0029" created.
>[kubeexec] DEBUG 2018/06/08 08:54:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_8546c963be0cfb030d57435e94a50d61 /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_8546c963be0cfb030d57435e94a50d61
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir -p /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_dddf81e8d43f425cc26134166b21f15e
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_8546c963be0cfb030d57435e94a50d61/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_cdc8c0d24f68026f06c17385d2fd0029
>Result: meta-data=/dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_cdc8c0d24f68026f06c17385d2fd0029 isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:54:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: chown :2004 /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_8546c963be0cfb030d57435e94a50d61/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: awk "BEGIN {print \"/dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_cdc8c0d24f68026f06c17385d2fd0029 /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_cdc8c0d24f68026f06c17385d2fd0029 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: chmod 2775 /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_8546c963be0cfb030d57435e94a50d61/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_cdc8c0d24f68026f06c17385d2fd0029 /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_cdc8c0d24f68026f06c17385d2fd0029
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_d389f0278a774bd7443a09af960961d8/tp_dddf81e8d43f425cc26134166b21f15e --virtualsize 2097152K --name brick_dddf81e8d43f425cc26134166b21f15e
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_dddf81e8d43f425cc26134166b21f15e" created.
>[kubeexec] DEBUG 2018/06/08 08:54:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_cdc8c0d24f68026f06c17385d2fd0029/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir -p /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_6319c734fbf95bbbfb1156b24dcc43ea
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_dddf81e8d43f425cc26134166b21f15e
>Result: meta-data=/dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_dddf81e8d43f425cc26134166b21f15e isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:54:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: chown :2005 /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_cdc8c0d24f68026f06c17385d2fd0029/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: awk "BEGIN {print \"/dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_dddf81e8d43f425cc26134166b21f15e /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_dddf81e8d43f425cc26134166b21f15e xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 91.519Âµs
>[kubeexec] DEBUG 2018/06/08 08:54:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: chmod 2775 /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_cdc8c0d24f68026f06c17385d2fd0029/brick
>Result: 
>[cmdexec] INFO 2018/06/08 08:54:34 Creating volume vol_1918777ef3ce84df17c8a114fb89f33e replica 3
>[kubeexec] DEBUG 2018/06/08 08:54:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_dddf81e8d43f425cc26134166b21f15e /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_dddf81e8d43f425cc26134166b21f15e
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_9394bc70699b006c5460c9f654cf345f/tp_6319c734fbf95bbbfb1156b24dcc43ea --virtualsize 2097152K --name brick_6319c734fbf95bbbfb1156b24dcc43ea
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_6319c734fbf95bbbfb1156b24dcc43ea" created.
>[kubeexec] DEBUG 2018/06/08 08:54:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_dddf81e8d43f425cc26134166b21f15e/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:35 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir -p /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_d985a824b73813069566b0a71bb73269
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:35 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_6319c734fbf95bbbfb1156b24dcc43ea
>Result: meta-data=/dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_6319c734fbf95bbbfb1156b24dcc43ea isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:54:35 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: chown :2007 /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_dddf81e8d43f425cc26134166b21f15e/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:35 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: awk "BEGIN {print \"/dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_6319c734fbf95bbbfb1156b24dcc43ea /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_6319c734fbf95bbbfb1156b24dcc43ea xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:35 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: chmod 2775 /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_dddf81e8d43f425cc26134166b21f15e/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:35 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_6319c734fbf95bbbfb1156b24dcc43ea /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_6319c734fbf95bbbfb1156b24dcc43ea
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:35 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_3a4297677881963e3f80124971d50eea/tp_d985a824b73813069566b0a71bb73269 --virtualsize 2097152K --name brick_d985a824b73813069566b0a71bb73269
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_d985a824b73813069566b0a71bb73269" created.
>[kubeexec] DEBUG 2018/06/08 08:54:35 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_6319c734fbf95bbbfb1156b24dcc43ea/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:35 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir -p /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_0a07a24688dfe13d4e1726b8d05e24a7
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:35 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_d985a824b73813069566b0a71bb73269
>Result: meta-data=/dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_d985a824b73813069566b0a71bb73269 isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:54:35 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: chown :2003 /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_6319c734fbf95bbbfb1156b24dcc43ea/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:35 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: awk "BEGIN {print \"/dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_d985a824b73813069566b0a71bb73269 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_d985a824b73813069566b0a71bb73269 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:35 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: chmod 2775 /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_6319c734fbf95bbbfb1156b24dcc43ea/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:35 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_d985a824b73813069566b0a71bb73269 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_d985a824b73813069566b0a71bb73269
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:35 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_b861af9cc67f5bf8d5a8f9e38f56b14e/tp_0a07a24688dfe13d4e1726b8d05e24a7 --virtualsize 2097152K --name brick_0a07a24688dfe13d4e1726b8d05e24a7
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_0a07a24688dfe13d4e1726b8d05e24a7" created.
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 235.352Âµs
>[kubeexec] DEBUG 2018/06/08 08:54:35 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_d985a824b73813069566b0a71bb73269/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:35 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir -p /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_322eeb9ae02c4f9e338567566394d2c7
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:35 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_0a07a24688dfe13d4e1726b8d05e24a7
>Result: meta-data=/dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_0a07a24688dfe13d4e1726b8d05e24a7 isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[negroni] Started DELETE /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Started DELETE /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 202 Accepted in 21.165555ms
>[asynchttp] INFO 2018/06/08 08:54:35 asynchttp.go:288: Started job cc7d3ffb84c36288d9c4c22cf55e3ada
>[heketi] INFO 2018/06/08 08:54:35 Started async operation: Delete Volume
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 88.54Âµs
>[negroni] Completed 202 Accepted in 27.561916ms
>[asynchttp] INFO 2018/06/08 08:54:35 asynchttp.go:288: Started job 30a446e1b3f0ca388030dee9d6823363
>[heketi] INFO 2018/06/08 08:54:35 Started async operation: Delete Volume
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 131.6Âµs
>[kubeexec] DEBUG 2018/06/08 08:54:35 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: chown :2007 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_d985a824b73813069566b0a71bb73269/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:35 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: awk "BEGIN {print \"/dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_0a07a24688dfe13d4e1726b8d05e24a7 /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_0a07a24688dfe13d4e1726b8d05e24a7 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: chmod 2775 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_d985a824b73813069566b0a71bb73269/brick
>Result: 
>[cmdexec] INFO 2018/06/08 08:54:36 Creating volume vol_afef5f4a84abec3ab3e4f2a5bae2db23 replica 3
>[kubeexec] DEBUG 2018/06/08 08:54:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_0a07a24688dfe13d4e1726b8d05e24a7 /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_0a07a24688dfe13d4e1726b8d05e24a7
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_9394bc70699b006c5460c9f654cf345f/tp_322eeb9ae02c4f9e338567566394d2c7 --virtualsize 2097152K --name brick_322eeb9ae02c4f9e338567566394d2c7
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_322eeb9ae02c4f9e338567566394d2c7" created.
>[kubeexec] DEBUG 2018/06/08 08:54:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_0a07a24688dfe13d4e1726b8d05e24a7/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir -p /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_bd4fcfec1db80add56b57e18bd681378
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: chown :2006 /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_0a07a24688dfe13d4e1726b8d05e24a7/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_322eeb9ae02c4f9e338567566394d2c7
>Result: meta-data=/dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_322eeb9ae02c4f9e338567566394d2c7 isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:54:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: chmod 2775 /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_0a07a24688dfe13d4e1726b8d05e24a7/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: awk "BEGIN {print \"/dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_322eeb9ae02c4f9e338567566394d2c7 /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_322eeb9ae02c4f9e338567566394d2c7 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_3a4297677881963e3f80124971d50eea/tp_bd4fcfec1db80add56b57e18bd681378 --virtualsize 2097152K --name brick_bd4fcfec1db80add56b57e18bd681378
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_bd4fcfec1db80add56b57e18bd681378" created.
>[kubeexec] DEBUG 2018/06/08 08:54:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_322eeb9ae02c4f9e338567566394d2c7 /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_322eeb9ae02c4f9e338567566394d2c7
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir -p /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_c740e50db586c869e27205aee2820dd7
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_322eeb9ae02c4f9e338567566394d2c7/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_bd4fcfec1db80add56b57e18bd681378
>Result: meta-data=/dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_bd4fcfec1db80add56b57e18bd681378 isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:54:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: chown :2007 /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_322eeb9ae02c4f9e338567566394d2c7/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: awk "BEGIN {print \"/dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_bd4fcfec1db80add56b57e18bd681378 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_bd4fcfec1db80add56b57e18bd681378 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 120.57Âµs
>[kubeexec] DEBUG 2018/06/08 08:54:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: chmod 2775 /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_322eeb9ae02c4f9e338567566394d2c7/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_bd4fcfec1db80add56b57e18bd681378 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_bd4fcfec1db80add56b57e18bd681378
>Result: 
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 114.311Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 147.951Âµs
>[kubeexec] DEBUG 2018/06/08 08:54:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_b861af9cc67f5bf8d5a8f9e38f56b14e/tp_c740e50db586c869e27205aee2820dd7 --virtualsize 2097152K --name brick_c740e50db586c869e27205aee2820dd7
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_c740e50db586c869e27205aee2820dd7" created.
>[kubeexec] DEBUG 2018/06/08 08:54:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_bd4fcfec1db80add56b57e18bd681378/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir -p /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_3ca7a79bfe1ee21131ef20732eac93d7
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_c740e50db586c869e27205aee2820dd7
>Result: meta-data=/dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_c740e50db586c869e27205aee2820dd7 isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:54:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: chown :2006 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_bd4fcfec1db80add56b57e18bd681378/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: awk "BEGIN {print \"/dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_c740e50db586c869e27205aee2820dd7 /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_c740e50db586c869e27205aee2820dd7 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: chmod 2775 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_bd4fcfec1db80add56b57e18bd681378/brick
>Result: 
>[cmdexec] INFO 2018/06/08 08:54:37 Creating volume vol_daf13b7f607d1b280c78e909af25a215 replica 3
>[kubeexec] DEBUG 2018/06/08 08:54:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_c740e50db586c869e27205aee2820dd7 /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_c740e50db586c869e27205aee2820dd7
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_9394bc70699b006c5460c9f654cf345f/tp_3ca7a79bfe1ee21131ef20732eac93d7 --virtualsize 2097152K --name brick_3ca7a79bfe1ee21131ef20732eac93d7
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_3ca7a79bfe1ee21131ef20732eac93d7" created.
>[kubeexec] DEBUG 2018/06/08 08:54:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir -p /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_f88d16c3bc0200c12aa9db817d4d9f0a
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_c740e50db586c869e27205aee2820dd7/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_3ca7a79bfe1ee21131ef20732eac93d7
>Result: meta-data=/dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_3ca7a79bfe1ee21131ef20732eac93d7 isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:54:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: chown :2001 /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_c740e50db586c869e27205aee2820dd7/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: awk "BEGIN {print \"/dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_3ca7a79bfe1ee21131ef20732eac93d7 /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_3ca7a79bfe1ee21131ef20732eac93d7 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: chmod 2775 /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_c740e50db586c869e27205aee2820dd7/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_3a4297677881963e3f80124971d50eea/tp_f88d16c3bc0200c12aa9db817d4d9f0a --virtualsize 2097152K --name brick_f88d16c3bc0200c12aa9db817d4d9f0a
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_f88d16c3bc0200c12aa9db817d4d9f0a" created.
>[kubeexec] DEBUG 2018/06/08 08:54:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_3ca7a79bfe1ee21131ef20732eac93d7 /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_3ca7a79bfe1ee21131ef20732eac93d7
>Result: 
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 126.576Âµs
>[kubeexec] DEBUG 2018/06/08 08:54:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir -p /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_b35881161d7bf25bfa4af1bea4746d58
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_3ca7a79bfe1ee21131ef20732eac93d7/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_f88d16c3bc0200c12aa9db817d4d9f0a
>Result: meta-data=/dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_f88d16c3bc0200c12aa9db817d4d9f0a isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 92.327Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 152.783Âµs
>[kubeexec] DEBUG 2018/06/08 08:54:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: awk "BEGIN {print \"/dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_f88d16c3bc0200c12aa9db817d4d9f0a /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_f88d16c3bc0200c12aa9db817d4d9f0a xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: chown :2006 /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_3ca7a79bfe1ee21131ef20732eac93d7/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: chmod 2775 /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_3ca7a79bfe1ee21131ef20732eac93d7/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_f88d16c3bc0200c12aa9db817d4d9f0a /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_f88d16c3bc0200c12aa9db817d4d9f0a
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_d389f0278a774bd7443a09af960961d8/tp_b35881161d7bf25bfa4af1bea4746d58 --virtualsize 2097152K --name brick_b35881161d7bf25bfa4af1bea4746d58
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_b35881161d7bf25bfa4af1bea4746d58" created.
>[kubeexec] DEBUG 2018/06/08 08:54:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir -p /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_30eb22cad2c3536d381b8d71943cffd2
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_f88d16c3bc0200c12aa9db817d4d9f0a/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_b35881161d7bf25bfa4af1bea4746d58
>Result: meta-data=/dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_b35881161d7bf25bfa4af1bea4746d58 isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:54:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: chown :2001 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_f88d16c3bc0200c12aa9db817d4d9f0a/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: awk "BEGIN {print \"/dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_b35881161d7bf25bfa4af1bea4746d58 /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_b35881161d7bf25bfa4af1bea4746d58 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: chmod 2775 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_f88d16c3bc0200c12aa9db817d4d9f0a/brick
>Result: 
>[cmdexec] INFO 2018/06/08 08:54:38 Creating volume vol_337bf2c01bf8c45eec5bab53ad5c2e46 replica 3
>[kubeexec] DEBUG 2018/06/08 08:54:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_b35881161d7bf25bfa4af1bea4746d58 /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_b35881161d7bf25bfa4af1bea4746d58
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_96f1667f2f1ced2c5ef94772922be93b/tp_30eb22cad2c3536d381b8d71943cffd2 --virtualsize 2097152K --name brick_30eb22cad2c3536d381b8d71943cffd2
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_30eb22cad2c3536d381b8d71943cffd2" created.
>[kubeexec] DEBUG 2018/06/08 08:54:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir -p /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_a84d4290cbebd984dd11b88eb806d93b
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_b35881161d7bf25bfa4af1bea4746d58/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_30eb22cad2c3536d381b8d71943cffd2
>Result: meta-data=/dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_30eb22cad2c3536d381b8d71943cffd2 isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:54:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: chown :2002 /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_b35881161d7bf25bfa4af1bea4746d58/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: awk "BEGIN {print \"/dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_30eb22cad2c3536d381b8d71943cffd2 /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_30eb22cad2c3536d381b8d71943cffd2 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 85.469Âµs
>[kubeexec] DEBUG 2018/06/08 08:54:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: chmod 2775 /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_b35881161d7bf25bfa4af1bea4746d58/brick
>Result: 
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 119.632Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 116.42Âµs
>[kubeexec] DEBUG 2018/06/08 08:54:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_30eb22cad2c3536d381b8d71943cffd2 /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_30eb22cad2c3536d381b8d71943cffd2
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_66a9af9f6bac95f9d8d556a2f14c29d3/tp_a84d4290cbebd984dd11b88eb806d93b --virtualsize 2097152K --name brick_a84d4290cbebd984dd11b88eb806d93b
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_a84d4290cbebd984dd11b88eb806d93b" created.
>[kubeexec] DEBUG 2018/06/08 08:54:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir -p /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_bf4ebc8a0c81383f3dc357f04e4903c1
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_30eb22cad2c3536d381b8d71943cffd2/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:39 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_a84d4290cbebd984dd11b88eb806d93b
>Result: meta-data=/dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_a84d4290cbebd984dd11b88eb806d93b isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:54:39 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: chown :2005 /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_30eb22cad2c3536d381b8d71943cffd2/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:39 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: awk "BEGIN {print \"/dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_a84d4290cbebd984dd11b88eb806d93b /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_a84d4290cbebd984dd11b88eb806d93b xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:39 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: chmod 2775 /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_30eb22cad2c3536d381b8d71943cffd2/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:39 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_a84d4290cbebd984dd11b88eb806d93b /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_a84d4290cbebd984dd11b88eb806d93b
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:39 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_b861af9cc67f5bf8d5a8f9e38f56b14e/tp_bf4ebc8a0c81383f3dc357f04e4903c1 --virtualsize 2097152K --name brick_bf4ebc8a0c81383f3dc357f04e4903c1
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_bf4ebc8a0c81383f3dc357f04e4903c1" created.
>[kubeexec] DEBUG 2018/06/08 08:54:39 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_a84d4290cbebd984dd11b88eb806d93b/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:39 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir -p /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_02dbc9628a1cd8a5f20d9fea7a615dcc
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:39 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_bf4ebc8a0c81383f3dc357f04e4903c1
>Result: meta-data=/dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_bf4ebc8a0c81383f3dc357f04e4903c1 isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:54:39 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: chown :2002 /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_a84d4290cbebd984dd11b88eb806d93b/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:39 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: awk "BEGIN {print \"/dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_bf4ebc8a0c81383f3dc357f04e4903c1 /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_bf4ebc8a0c81383f3dc357f04e4903c1 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:39 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: chmod 2775 /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_a84d4290cbebd984dd11b88eb806d93b/brick
>Result: 
>[cmdexec] INFO 2018/06/08 08:54:39 Creating volume vol_f73270331b95278f490fd1dfe0b010df replica 3
>[kubeexec] DEBUG 2018/06/08 08:54:39 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_bf4ebc8a0c81383f3dc357f04e4903c1 /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_bf4ebc8a0c81383f3dc357f04e4903c1
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:39 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_96f1667f2f1ced2c5ef94772922be93b/tp_02dbc9628a1cd8a5f20d9fea7a615dcc --virtualsize 2097152K --name brick_02dbc9628a1cd8a5f20d9fea7a615dcc
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_02dbc9628a1cd8a5f20d9fea7a615dcc" created.
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 105.998Âµs
>[kubeexec] DEBUG 2018/06/08 08:54:39 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_bf4ebc8a0c81383f3dc357f04e4903c1/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:39 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir -p /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_2bba219914cb29c5994021b1731b7763
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:39 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_02dbc9628a1cd8a5f20d9fea7a615dcc
>Result: meta-data=/dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_02dbc9628a1cd8a5f20d9fea7a615dcc isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 116.679Âµs
>[kubeexec] DEBUG 2018/06/08 08:54:39 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: chown :2004 /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_bf4ebc8a0c81383f3dc357f04e4903c1/brick
>Result: 
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 83.609Âµs
>[kubeexec] DEBUG 2018/06/08 08:54:39 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: awk "BEGIN {print \"/dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_02dbc9628a1cd8a5f20d9fea7a615dcc /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_02dbc9628a1cd8a5f20d9fea7a615dcc xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:39 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: chmod 2775 /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_bf4ebc8a0c81383f3dc357f04e4903c1/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:40 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_02dbc9628a1cd8a5f20d9fea7a615dcc /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_02dbc9628a1cd8a5f20d9fea7a615dcc
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:40 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_66a9af9f6bac95f9d8d556a2f14c29d3/tp_2bba219914cb29c5994021b1731b7763 --virtualsize 2097152K --name brick_2bba219914cb29c5994021b1731b7763
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_2bba219914cb29c5994021b1731b7763" created.
>[kubeexec] DEBUG 2018/06/08 08:54:40 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_02dbc9628a1cd8a5f20d9fea7a615dcc/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:40 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir -p /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_6167bec47d9258cf5b2a5e3a48c0d391
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:40 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_2bba219914cb29c5994021b1731b7763
>Result: meta-data=/dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_2bba219914cb29c5994021b1731b7763 isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:54:40 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: chown :2001 /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_02dbc9628a1cd8a5f20d9fea7a615dcc/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:40 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: awk "BEGIN {print \"/dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_2bba219914cb29c5994021b1731b7763 /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_2bba219914cb29c5994021b1731b7763 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:40 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: chmod 2775 /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_02dbc9628a1cd8a5f20d9fea7a615dcc/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:40 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_2bba219914cb29c5994021b1731b7763 /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_2bba219914cb29c5994021b1731b7763
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:40 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_b861af9cc67f5bf8d5a8f9e38f56b14e/tp_6167bec47d9258cf5b2a5e3a48c0d391 --virtualsize 2097152K --name brick_6167bec47d9258cf5b2a5e3a48c0d391
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_6167bec47d9258cf5b2a5e3a48c0d391" created.
>[kubeexec] DEBUG 2018/06/08 08:54:40 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir -p /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_dde70bf2e86fac7df0e0427af9bf5db3
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:40 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_2bba219914cb29c5994021b1731b7763/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:40 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_6167bec47d9258cf5b2a5e3a48c0d391
>Result: meta-data=/dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_6167bec47d9258cf5b2a5e3a48c0d391 isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:54:40 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: chown :2004 /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_2bba219914cb29c5994021b1731b7763/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:40 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: awk "BEGIN {print \"/dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_6167bec47d9258cf5b2a5e3a48c0d391 /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_6167bec47d9258cf5b2a5e3a48c0d391 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 238.953Âµs
>[kubeexec] DEBUG 2018/06/08 08:54:40 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: chmod 2775 /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_2bba219914cb29c5994021b1731b7763/brick
>Result: 
>[cmdexec] INFO 2018/06/08 08:54:40 Creating volume vol_1ef58be42cf8ea7cf9298cff303e903a replica 3
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 97.468Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 109.285Âµs
>[kubeexec] DEBUG 2018/06/08 08:54:40 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_6167bec47d9258cf5b2a5e3a48c0d391 /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_6167bec47d9258cf5b2a5e3a48c0d391
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:40 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_9394bc70699b006c5460c9f654cf345f/tp_dde70bf2e86fac7df0e0427af9bf5db3 --virtualsize 2097152K --name brick_dde70bf2e86fac7df0e0427af9bf5db3
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_dde70bf2e86fac7df0e0427af9bf5db3" created.
>[kubeexec] DEBUG 2018/06/08 08:54:41 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir -p /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_b2827681c0e1655380b4d5d53226bf7e
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:41 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_6167bec47d9258cf5b2a5e3a48c0d391/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:41 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_dde70bf2e86fac7df0e0427af9bf5db3
>Result: meta-data=/dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_dde70bf2e86fac7df0e0427af9bf5db3 isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:54:41 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: chown :2003 /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_6167bec47d9258cf5b2a5e3a48c0d391/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:41 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: awk "BEGIN {print \"/dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_dde70bf2e86fac7df0e0427af9bf5db3 /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_dde70bf2e86fac7df0e0427af9bf5db3 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:41 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: chmod 2775 /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_6167bec47d9258cf5b2a5e3a48c0d391/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:41 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_dde70bf2e86fac7df0e0427af9bf5db3 /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_dde70bf2e86fac7df0e0427af9bf5db3
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:41 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_66a9af9f6bac95f9d8d556a2f14c29d3/tp_b2827681c0e1655380b4d5d53226bf7e --virtualsize 2097152K --name brick_b2827681c0e1655380b4d5d53226bf7e
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_b2827681c0e1655380b4d5d53226bf7e" created.
>[kubeexec] DEBUG 2018/06/08 08:54:41 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir -p /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_9963c5d95158f58c8fc2f28d81d925ea
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:41 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_dde70bf2e86fac7df0e0427af9bf5db3/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:41 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_b2827681c0e1655380b4d5d53226bf7e
>Result: meta-data=/dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_b2827681c0e1655380b4d5d53226bf7e isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:54:41 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: chown :2002 /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_dde70bf2e86fac7df0e0427af9bf5db3/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:41 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: awk "BEGIN {print \"/dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_b2827681c0e1655380b4d5d53226bf7e /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_b2827681c0e1655380b4d5d53226bf7e xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:41 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: chmod 2775 /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_dde70bf2e86fac7df0e0427af9bf5db3/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:41 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_b2827681c0e1655380b4d5d53226bf7e /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_b2827681c0e1655380b4d5d53226bf7e
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:41 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_b861af9cc67f5bf8d5a8f9e38f56b14e/tp_9963c5d95158f58c8fc2f28d81d925ea --virtualsize 2097152K --name brick_9963c5d95158f58c8fc2f28d81d925ea
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_9963c5d95158f58c8fc2f28d81d925ea" created.
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 135.223Âµs
>[kubeexec] DEBUG 2018/06/08 08:54:41 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir -p /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_da658ca7b4961a716b886b57a67ff000
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:41 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_b2827681c0e1655380b4d5d53226bf7e/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:41 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_9963c5d95158f58c8fc2f28d81d925ea
>Result: meta-data=/dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_9963c5d95158f58c8fc2f28d81d925ea isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 155.571Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 220.734Âµs
>[kubeexec] DEBUG 2018/06/08 08:54:41 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: chown :2003 /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_b2827681c0e1655380b4d5d53226bf7e/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:41 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: awk "BEGIN {print \"/dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_9963c5d95158f58c8fc2f28d81d925ea /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_9963c5d95158f58c8fc2f28d81d925ea xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:42 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: chmod 2775 /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_b2827681c0e1655380b4d5d53226bf7e/brick
>Result: 
>[cmdexec] INFO 2018/06/08 08:54:42 Creating volume vol_1a828ff2310b778d09ffdadd755dc5ee replica 3
>[kubeexec] DEBUG 2018/06/08 08:54:42 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_9963c5d95158f58c8fc2f28d81d925ea /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_9963c5d95158f58c8fc2f28d81d925ea
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:42 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_9394bc70699b006c5460c9f654cf345f/tp_da658ca7b4961a716b886b57a67ff000 --virtualsize 2097152K --name brick_da658ca7b4961a716b886b57a67ff000
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_da658ca7b4961a716b886b57a67ff000" created.
>[kubeexec] DEBUG 2018/06/08 08:54:42 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_9963c5d95158f58c8fc2f28d81d925ea/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:42 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir -p /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_a1150dfd2cad12bad69ef9cc830da83d
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:42 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_da658ca7b4961a716b886b57a67ff000
>Result: meta-data=/dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_da658ca7b4961a716b886b57a67ff000 isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:54:42 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: chown :2007 /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_9963c5d95158f58c8fc2f28d81d925ea/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:42 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: awk "BEGIN {print \"/dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_da658ca7b4961a716b886b57a67ff000 /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_da658ca7b4961a716b886b57a67ff000 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:42 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: chmod 2775 /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_9963c5d95158f58c8fc2f28d81d925ea/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:42 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_da658ca7b4961a716b886b57a67ff000 /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_da658ca7b4961a716b886b57a67ff000
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:42 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_3a4297677881963e3f80124971d50eea/tp_a1150dfd2cad12bad69ef9cc830da83d --virtualsize 2097152K --name brick_a1150dfd2cad12bad69ef9cc830da83d
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_a1150dfd2cad12bad69ef9cc830da83d" created.
>[kubeexec] DEBUG 2018/06/08 08:54:42 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir -p /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_4d4160003ee228514a59c2c61932ceed
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:42 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_da658ca7b4961a716b886b57a67ff000/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:42 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_a1150dfd2cad12bad69ef9cc830da83d
>Result: meta-data=/dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_a1150dfd2cad12bad69ef9cc830da83d isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:54:42 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: chown :2003 /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_da658ca7b4961a716b886b57a67ff000/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:42 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: awk "BEGIN {print \"/dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_a1150dfd2cad12bad69ef9cc830da83d /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_a1150dfd2cad12bad69ef9cc830da83d xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 101.395Âµs
>[kubeexec] DEBUG 2018/06/08 08:54:42 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: chmod 2775 /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_da658ca7b4961a716b886b57a67ff000/brick
>Result: 
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 116.084Âµs
>[kubeexec] DEBUG 2018/06/08 08:54:42 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_a1150dfd2cad12bad69ef9cc830da83d /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_a1150dfd2cad12bad69ef9cc830da83d
>Result: 
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 204.43Âµs
>[kubeexec] DEBUG 2018/06/08 08:54:42 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_d389f0278a774bd7443a09af960961d8/tp_4d4160003ee228514a59c2c61932ceed --virtualsize 2097152K --name brick_4d4160003ee228514a59c2c61932ceed
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_4d4160003ee228514a59c2c61932ceed" created.
>[kubeexec] DEBUG 2018/06/08 08:54:43 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_a1150dfd2cad12bad69ef9cc830da83d/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:43 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir -p /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_0a39774f01ac7959661ff5afff2595ab
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:43 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_4d4160003ee228514a59c2c61932ceed
>Result: meta-data=/dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_4d4160003ee228514a59c2c61932ceed isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:54:43 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: chown :2007 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_a1150dfd2cad12bad69ef9cc830da83d/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:43 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: awk "BEGIN {print \"/dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_4d4160003ee228514a59c2c61932ceed /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_4d4160003ee228514a59c2c61932ceed xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:43 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: chmod 2775 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_a1150dfd2cad12bad69ef9cc830da83d/brick
>Result: 
>[cmdexec] INFO 2018/06/08 08:54:43 Creating volume vol_9fb2830da79dd70d910dad8426dc236f replica 3
>[kubeexec] DEBUG 2018/06/08 08:54:43 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_4d4160003ee228514a59c2c61932ceed /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_4d4160003ee228514a59c2c61932ceed
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:43 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_4d4160003ee228514a59c2c61932ceed/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:43 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir -p /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_211fca2005300a58cf3b464b89e9deea
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:43 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_9394bc70699b006c5460c9f654cf345f/tp_0a39774f01ac7959661ff5afff2595ab --virtualsize 2097152K --name brick_0a39774f01ac7959661ff5afff2595ab
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_0a39774f01ac7959661ff5afff2595ab" created.
>[kubeexec] DEBUG 2018/06/08 08:54:43 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: chown :2006 /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_4d4160003ee228514a59c2c61932ceed/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:43 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_0a39774f01ac7959661ff5afff2595ab
>Result: meta-data=/dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_0a39774f01ac7959661ff5afff2595ab isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:54:43 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: chmod 2775 /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_4d4160003ee228514a59c2c61932ceed/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:43 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: awk "BEGIN {print \"/dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_0a39774f01ac7959661ff5afff2595ab /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_0a39774f01ac7959661ff5afff2595ab xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:43 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_3a4297677881963e3f80124971d50eea/tp_211fca2005300a58cf3b464b89e9deea --virtualsize 2097152K --name brick_211fca2005300a58cf3b464b89e9deea
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_211fca2005300a58cf3b464b89e9deea" created.
>[kubeexec] DEBUG 2018/06/08 08:54:43 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_0a39774f01ac7959661ff5afff2595ab /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_0a39774f01ac7959661ff5afff2595ab
>Result: 
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 121.029Âµs
>[kubeexec] DEBUG 2018/06/08 08:54:43 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir -p /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_df153caad54b6511504d5af0cdc4e2b5
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:43 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_211fca2005300a58cf3b464b89e9deea
>Result: meta-data=/dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_211fca2005300a58cf3b464b89e9deea isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:54:43 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_0a39774f01ac7959661ff5afff2595ab/brick
>Result: 
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 114.319Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 108.827Âµs
>[kubeexec] DEBUG 2018/06/08 08:54:43 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: awk "BEGIN {print \"/dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_211fca2005300a58cf3b464b89e9deea /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_211fca2005300a58cf3b464b89e9deea xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:43 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: chown :2004 /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_0a39774f01ac7959661ff5afff2595ab/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:44 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: chmod 2775 /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_0a39774f01ac7959661ff5afff2595ab/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:44 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_211fca2005300a58cf3b464b89e9deea /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_211fca2005300a58cf3b464b89e9deea
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:44 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_b861af9cc67f5bf8d5a8f9e38f56b14e/tp_df153caad54b6511504d5af0cdc4e2b5 --virtualsize 2097152K --name brick_df153caad54b6511504d5af0cdc4e2b5
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_df153caad54b6511504d5af0cdc4e2b5" created.
>[kubeexec] DEBUG 2018/06/08 08:54:44 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir -p /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_c8dd6b39719a6bc75ea331fae5a92396
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:44 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_211fca2005300a58cf3b464b89e9deea/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:44 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_df153caad54b6511504d5af0cdc4e2b5
>Result: meta-data=/dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_df153caad54b6511504d5af0cdc4e2b5 isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:54:44 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: chown :2006 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_211fca2005300a58cf3b464b89e9deea/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:44 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: awk "BEGIN {print \"/dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_df153caad54b6511504d5af0cdc4e2b5 /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_df153caad54b6511504d5af0cdc4e2b5 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:44 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: chmod 2775 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_211fca2005300a58cf3b464b89e9deea/brick
>Result: 
>[cmdexec] INFO 2018/06/08 08:54:44 Creating volume vol_dc9ab13a25ccbad8262fba92766a31f9 replica 3
>[kubeexec] DEBUG 2018/06/08 08:54:44 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_9394bc70699b006c5460c9f654cf345f/tp_c8dd6b39719a6bc75ea331fae5a92396 --virtualsize 2097152K --name brick_c8dd6b39719a6bc75ea331fae5a92396
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_c8dd6b39719a6bc75ea331fae5a92396" created.
>[kubeexec] DEBUG 2018/06/08 08:54:44 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_df153caad54b6511504d5af0cdc4e2b5 /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_df153caad54b6511504d5af0cdc4e2b5
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:44 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir -p /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_54258f15bd3df0dfb8421196bdea9774
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:44 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_df153caad54b6511504d5af0cdc4e2b5/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:44 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_c8dd6b39719a6bc75ea331fae5a92396
>Result: meta-data=/dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_c8dd6b39719a6bc75ea331fae5a92396 isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:54:44 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: chown :2005 /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_df153caad54b6511504d5af0cdc4e2b5/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:44 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: awk "BEGIN {print \"/dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_c8dd6b39719a6bc75ea331fae5a92396 /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_c8dd6b39719a6bc75ea331fae5a92396 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 112.912Âµs
>[kubeexec] DEBUG 2018/06/08 08:54:44 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: chmod 2775 /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_df153caad54b6511504d5af0cdc4e2b5/brick
>Result: 
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 110.642Âµs
>[kubeexec] DEBUG 2018/06/08 08:54:44 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_c8dd6b39719a6bc75ea331fae5a92396 /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_c8dd6b39719a6bc75ea331fae5a92396
>Result: 
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 121.123Âµs
>[kubeexec] DEBUG 2018/06/08 08:54:44 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_3a4297677881963e3f80124971d50eea/tp_54258f15bd3df0dfb8421196bdea9774 --virtualsize 2097152K --name brick_54258f15bd3df0dfb8421196bdea9774
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_54258f15bd3df0dfb8421196bdea9774" created.
>[kubeexec] DEBUG 2018/06/08 08:54:44 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_c8dd6b39719a6bc75ea331fae5a92396/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir -p /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_e1709c1db8499b885817e5f4682c56ca
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_54258f15bd3df0dfb8421196bdea9774
>Result: meta-data=/dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_54258f15bd3df0dfb8421196bdea9774 isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:54:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: chown :2005 /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_c8dd6b39719a6bc75ea331fae5a92396/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: awk "BEGIN {print \"/dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_54258f15bd3df0dfb8421196bdea9774 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_54258f15bd3df0dfb8421196bdea9774 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: chmod 2775 /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_c8dd6b39719a6bc75ea331fae5a92396/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_54258f15bd3df0dfb8421196bdea9774 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_54258f15bd3df0dfb8421196bdea9774
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_b861af9cc67f5bf8d5a8f9e38f56b14e/tp_e1709c1db8499b885817e5f4682c56ca --virtualsize 2097152K --name brick_e1709c1db8499b885817e5f4682c56ca
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_e1709c1db8499b885817e5f4682c56ca" created.
>[kubeexec] DEBUG 2018/06/08 08:54:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_54258f15bd3df0dfb8421196bdea9774/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir -p /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_e54cabdc56fef4e5b4a11c7a72eaff3d
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_e1709c1db8499b885817e5f4682c56ca
>Result: meta-data=/dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_e1709c1db8499b885817e5f4682c56ca isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:54:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: chown :2005 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_54258f15bd3df0dfb8421196bdea9774/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: chmod 2775 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_54258f15bd3df0dfb8421196bdea9774/brick
>Result: 
>[cmdexec] INFO 2018/06/08 08:54:45 Creating volume vol_b7fb8f19c77039982909b8868f7af4cc replica 3
>[kubeexec] DEBUG 2018/06/08 08:54:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: awk "BEGIN {print \"/dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_e1709c1db8499b885817e5f4682c56ca /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_e1709c1db8499b885817e5f4682c56ca xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_e1709c1db8499b885817e5f4682c56ca /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_e1709c1db8499b885817e5f4682c56ca
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_96f1667f2f1ced2c5ef94772922be93b/tp_e54cabdc56fef4e5b4a11c7a72eaff3d --virtualsize 2097152K --name brick_e54cabdc56fef4e5b4a11c7a72eaff3d
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_e54cabdc56fef4e5b4a11c7a72eaff3d" created.
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 124.051Âµs
>[kubeexec] DEBUG 2018/06/08 08:54:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir -p /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_a69c00298f148d2208ea7eab0903a3d2
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_e1709c1db8499b885817e5f4682c56ca/brick
>Result: 
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 111.668Âµs
>[kubeexec] DEBUG 2018/06/08 08:54:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_e54cabdc56fef4e5b4a11c7a72eaff3d
>Result: meta-data=/dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_e54cabdc56fef4e5b4a11c7a72eaff3d isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 151.172Âµs
>[kubeexec] DEBUG 2018/06/08 08:54:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: chown :2001 /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_e1709c1db8499b885817e5f4682c56ca/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: awk "BEGIN {print \"/dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_e54cabdc56fef4e5b4a11c7a72eaff3d /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_e54cabdc56fef4e5b4a11c7a72eaff3d xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: chmod 2775 /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_e1709c1db8499b885817e5f4682c56ca/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_3a4297677881963e3f80124971d50eea/tp_a69c00298f148d2208ea7eab0903a3d2 --virtualsize 2097152K --name brick_a69c00298f148d2208ea7eab0903a3d2
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_a69c00298f148d2208ea7eab0903a3d2" created.
>[kubeexec] DEBUG 2018/06/08 08:54:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_e54cabdc56fef4e5b4a11c7a72eaff3d /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_e54cabdc56fef4e5b4a11c7a72eaff3d
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir -p /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_2a97d96e94e0ff7c49d2b6d81dfbd8fd
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_e54cabdc56fef4e5b4a11c7a72eaff3d/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_a69c00298f148d2208ea7eab0903a3d2
>Result: meta-data=/dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_a69c00298f148d2208ea7eab0903a3d2 isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:54:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: chown :2006 /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_e54cabdc56fef4e5b4a11c7a72eaff3d/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: awk "BEGIN {print \"/dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_a69c00298f148d2208ea7eab0903a3d2 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_a69c00298f148d2208ea7eab0903a3d2 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: chmod 2775 /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_e54cabdc56fef4e5b4a11c7a72eaff3d/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_d389f0278a774bd7443a09af960961d8/tp_2a97d96e94e0ff7c49d2b6d81dfbd8fd --virtualsize 2097152K --name brick_2a97d96e94e0ff7c49d2b6d81dfbd8fd
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_2a97d96e94e0ff7c49d2b6d81dfbd8fd" created.
>[kubeexec] DEBUG 2018/06/08 08:54:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_a69c00298f148d2208ea7eab0903a3d2 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_a69c00298f148d2208ea7eab0903a3d2
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir -p /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_68704fc0eb854fe2a0157b0261302792
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_a69c00298f148d2208ea7eab0903a3d2/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_2a97d96e94e0ff7c49d2b6d81dfbd8fd
>Result: meta-data=/dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_2a97d96e94e0ff7c49d2b6d81dfbd8fd isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:54:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: chown :2001 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_a69c00298f148d2208ea7eab0903a3d2/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: awk "BEGIN {print \"/dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_2a97d96e94e0ff7c49d2b6d81dfbd8fd /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_2a97d96e94e0ff7c49d2b6d81dfbd8fd xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 162.802Âµs
>[kubeexec] DEBUG 2018/06/08 08:54:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: chmod 2775 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_a69c00298f148d2208ea7eab0903a3d2/brick
>Result: 
>[cmdexec] INFO 2018/06/08 08:54:46 Creating volume vol_e25438438fd2d50a0b07f26b4bfb338a replica 3
>[kubeexec] DEBUG 2018/06/08 08:54:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_2a97d96e94e0ff7c49d2b6d81dfbd8fd /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_2a97d96e94e0ff7c49d2b6d81dfbd8fd
>Result: 
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 80.972Âµs
>[kubeexec] DEBUG 2018/06/08 08:54:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_96f1667f2f1ced2c5ef94772922be93b/tp_68704fc0eb854fe2a0157b0261302792 --virtualsize 2097152K --name brick_68704fc0eb854fe2a0157b0261302792
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_68704fc0eb854fe2a0157b0261302792" created.
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 340.123Âµs
>[kubeexec] DEBUG 2018/06/08 08:54:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_2a97d96e94e0ff7c49d2b6d81dfbd8fd/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir -p /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_b4e462e2cc3dbccfd86e44f907dc7f00
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:47 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_68704fc0eb854fe2a0157b0261302792
>Result: meta-data=/dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_68704fc0eb854fe2a0157b0261302792 isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:54:47 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: chown :2002 /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_2a97d96e94e0ff7c49d2b6d81dfbd8fd/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:47 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: awk "BEGIN {print \"/dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_68704fc0eb854fe2a0157b0261302792 /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_68704fc0eb854fe2a0157b0261302792 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:47 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: chmod 2775 /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_2a97d96e94e0ff7c49d2b6d81dfbd8fd/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:47 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_68704fc0eb854fe2a0157b0261302792 /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_68704fc0eb854fe2a0157b0261302792
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:47 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_3a4297677881963e3f80124971d50eea/tp_b4e462e2cc3dbccfd86e44f907dc7f00 --virtualsize 2097152K --name brick_b4e462e2cc3dbccfd86e44f907dc7f00
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_b4e462e2cc3dbccfd86e44f907dc7f00" created.
>[kubeexec] DEBUG 2018/06/08 08:54:47 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_68704fc0eb854fe2a0157b0261302792/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:47 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir -p /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_906b174d0f419aa3d1a9affe4675674a
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:47 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_b4e462e2cc3dbccfd86e44f907dc7f00
>Result: meta-data=/dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_b4e462e2cc3dbccfd86e44f907dc7f00 isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:54:47 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: chown :2007 /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_68704fc0eb854fe2a0157b0261302792/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:47 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: awk "BEGIN {print \"/dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_b4e462e2cc3dbccfd86e44f907dc7f00 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_b4e462e2cc3dbccfd86e44f907dc7f00 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:47 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: chmod 2775 /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_68704fc0eb854fe2a0157b0261302792/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:47 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_d389f0278a774bd7443a09af960961d8/tp_906b174d0f419aa3d1a9affe4675674a --virtualsize 2097152K --name brick_906b174d0f419aa3d1a9affe4675674a
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_906b174d0f419aa3d1a9affe4675674a" created.
>[kubeexec] DEBUG 2018/06/08 08:54:47 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_b4e462e2cc3dbccfd86e44f907dc7f00 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_b4e462e2cc3dbccfd86e44f907dc7f00
>Result: 
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 122.165Âµs
>[kubeexec] DEBUG 2018/06/08 08:54:47 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir -p /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_ad542929ce4fd8719fbe5fc44df98dbd
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:47 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_b4e462e2cc3dbccfd86e44f907dc7f00/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:47 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_906b174d0f419aa3d1a9affe4675674a
>Result: meta-data=/dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_906b174d0f419aa3d1a9affe4675674a isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 113.23Âµs
>[kubeexec] DEBUG 2018/06/08 08:54:47 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: chown :2002 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_b4e462e2cc3dbccfd86e44f907dc7f00/brick
>Result: 
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 399.463Âµs
>[kubeexec] DEBUG 2018/06/08 08:54:47 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: awk "BEGIN {print \"/dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_906b174d0f419aa3d1a9affe4675674a /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_906b174d0f419aa3d1a9affe4675674a xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:48 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: chmod 2775 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_b4e462e2cc3dbccfd86e44f907dc7f00/brick
>Result: 
>[cmdexec] INFO 2018/06/08 08:54:48 Creating volume vol_d21ed39ae095af2674175a798a0cb02c replica 3
>[kubeexec] DEBUG 2018/06/08 08:54:48 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_906b174d0f419aa3d1a9affe4675674a /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_906b174d0f419aa3d1a9affe4675674a
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:48 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvcreate --autobackup=n --poolmetadatasize 8192K --chunksize 256K --size 1048576K --thin vg_9394bc70699b006c5460c9f654cf345f/tp_ad542929ce4fd8719fbe5fc44df98dbd --virtualsize 1048576K --name brick_ad542929ce4fd8719fbe5fc44df98dbd
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_ad542929ce4fd8719fbe5fc44df98dbd" created.
>[kubeexec] DEBUG 2018/06/08 08:54:48 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_906b174d0f419aa3d1a9affe4675674a/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:48 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir -p /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_d1e261647e36a4dd323508df5b3decff
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:48 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_ad542929ce4fd8719fbe5fc44df98dbd
>Result: meta-data=/dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_ad542929ce4fd8719fbe5fc44df98dbd isize=512    agcount=8, agsize=32768 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=262144, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:54:48 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: chown :2003 /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_906b174d0f419aa3d1a9affe4675674a/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:48 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: awk "BEGIN {print \"/dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_ad542929ce4fd8719fbe5fc44df98dbd /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_ad542929ce4fd8719fbe5fc44df98dbd xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:48 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: chmod 2775 /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_906b174d0f419aa3d1a9affe4675674a/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:48 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_ad542929ce4fd8719fbe5fc44df98dbd /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_ad542929ce4fd8719fbe5fc44df98dbd
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:48 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_3a4297677881963e3f80124971d50eea/tp_d1e261647e36a4dd323508df5b3decff --virtualsize 2097152K --name brick_d1e261647e36a4dd323508df5b3decff
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_d1e261647e36a4dd323508df5b3decff" created.
>[kubeexec] DEBUG 2018/06/08 08:54:48 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir -p /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_592a78c302b01c3ee9538269481405e7
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:48 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_ad542929ce4fd8719fbe5fc44df98dbd/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:48 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_d1e261647e36a4dd323508df5b3decff
>Result: meta-data=/dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_d1e261647e36a4dd323508df5b3decff isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 117.729Âµs
>[kubeexec] DEBUG 2018/06/08 08:54:48 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: awk "BEGIN {print \"/dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_d1e261647e36a4dd323508df5b3decff /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_d1e261647e36a4dd323508df5b3decff xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 137.957Âµs
>[kubeexec] DEBUG 2018/06/08 08:54:48 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_b861af9cc67f5bf8d5a8f9e38f56b14e/tp_592a78c302b01c3ee9538269481405e7 --virtualsize 2097152K --name brick_592a78c302b01c3ee9538269481405e7
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_592a78c302b01c3ee9538269481405e7" created.
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 97.257Âµs
>[kubeexec] DEBUG 2018/06/08 08:54:48 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_d1e261647e36a4dd323508df5b3decff /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_d1e261647e36a4dd323508df5b3decff
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:48 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir -p /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_1039ed81d7d98ea183e9c7d8d00c1b6d
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:49 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_592a78c302b01c3ee9538269481405e7
>Result: meta-data=/dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_592a78c302b01c3ee9538269481405e7 isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:54:49 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_d1e261647e36a4dd323508df5b3decff/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:49 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: awk "BEGIN {print \"/dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_592a78c302b01c3ee9538269481405e7 /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_592a78c302b01c3ee9538269481405e7 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:49 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: chown :2003 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_d1e261647e36a4dd323508df5b3decff/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:49 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_592a78c302b01c3ee9538269481405e7 /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_592a78c302b01c3ee9538269481405e7
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:49 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: chmod 2775 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_d1e261647e36a4dd323508df5b3decff/brick
>Result: 
>[cmdexec] INFO 2018/06/08 08:54:49 Creating volume vol_21481d8911fe8ec238d97d71c1aa5cb3 replica 3
>[kubeexec] DEBUG 2018/06/08 08:54:49 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvcreate --autobackup=n --poolmetadatasize 8192K --chunksize 256K --size 1048576K --thin vg_9394bc70699b006c5460c9f654cf345f/tp_1039ed81d7d98ea183e9c7d8d00c1b6d --virtualsize 1048576K --name brick_1039ed81d7d98ea183e9c7d8d00c1b6d
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_1039ed81d7d98ea183e9c7d8d00c1b6d" created.
>[kubeexec] DEBUG 2018/06/08 08:54:49 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_592a78c302b01c3ee9538269481405e7/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:49 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir -p /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_11a59fea08d9b732822c2669ddaf54fa
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:49 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: chown :2004 /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_592a78c302b01c3ee9538269481405e7/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:49 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_1039ed81d7d98ea183e9c7d8d00c1b6d
>Result: meta-data=/dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_1039ed81d7d98ea183e9c7d8d00c1b6d isize=512    agcount=8, agsize=32768 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=262144, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:54:49 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: chmod 2775 /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_592a78c302b01c3ee9538269481405e7/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:49 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: awk "BEGIN {print \"/dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_1039ed81d7d98ea183e9c7d8d00c1b6d /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_1039ed81d7d98ea183e9c7d8d00c1b6d xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:49 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_1039ed81d7d98ea183e9c7d8d00c1b6d /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_1039ed81d7d98ea183e9c7d8d00c1b6d
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:49 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_66a9af9f6bac95f9d8d556a2f14c29d3/tp_11a59fea08d9b732822c2669ddaf54fa --virtualsize 2097152K --name brick_11a59fea08d9b732822c2669ddaf54fa
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_11a59fea08d9b732822c2669ddaf54fa" created.
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 152.692Âµs
>[kubeexec] DEBUG 2018/06/08 08:54:49 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir -p /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_dba8827a8c1b642d1b34bc5cf35aa4b4
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:49 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_1039ed81d7d98ea183e9c7d8d00c1b6d/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:49 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_11a59fea08d9b732822c2669ddaf54fa
>Result: meta-data=/dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_11a59fea08d9b732822c2669ddaf54fa isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 139.03Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 110.711Âµs
>[kubeexec] DEBUG 2018/06/08 08:54:49 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: awk "BEGIN {print \"/dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_11a59fea08d9b732822c2669ddaf54fa /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_11a59fea08d9b732822c2669ddaf54fa xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:50 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_11a59fea08d9b732822c2669ddaf54fa /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_11a59fea08d9b732822c2669ddaf54fa
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:50 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_d389f0278a774bd7443a09af960961d8/tp_dba8827a8c1b642d1b34bc5cf35aa4b4 --virtualsize 2097152K --name brick_dba8827a8c1b642d1b34bc5cf35aa4b4
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_dba8827a8c1b642d1b34bc5cf35aa4b4" created.
>[kubeexec] DEBUG 2018/06/08 08:54:50 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_11a59fea08d9b732822c2669ddaf54fa/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:50 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir -p /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_e1c7bf2add37671886c54c55f98f9fb7
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:50 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_dba8827a8c1b642d1b34bc5cf35aa4b4
>Result: meta-data=/dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_dba8827a8c1b642d1b34bc5cf35aa4b4 isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:54:50 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: chown :2004 /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_11a59fea08d9b732822c2669ddaf54fa/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:50 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: awk "BEGIN {print \"/dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_dba8827a8c1b642d1b34bc5cf35aa4b4 /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_dba8827a8c1b642d1b34bc5cf35aa4b4 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:50 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: chmod 2775 /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_11a59fea08d9b732822c2669ddaf54fa/brick
>Result: 
>[cmdexec] INFO 2018/06/08 08:54:50 Creating volume vol_226838416791f3286fcacb7e5f1ff59d replica 3
>[kubeexec] DEBUG 2018/06/08 08:54:50 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_dba8827a8c1b642d1b34bc5cf35aa4b4 /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_dba8827a8c1b642d1b34bc5cf35aa4b4
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:50 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvcreate --autobackup=n --poolmetadatasize 8192K --chunksize 256K --size 1048576K --thin vg_96f1667f2f1ced2c5ef94772922be93b/tp_e1c7bf2add37671886c54c55f98f9fb7 --virtualsize 1048576K --name brick_e1c7bf2add37671886c54c55f98f9fb7
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_e1c7bf2add37671886c54c55f98f9fb7" created.
>[kubeexec] DEBUG 2018/06/08 08:54:50 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir -p /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_b3918aae89c9aab7c5cfa3496f95936a
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:50 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_dba8827a8c1b642d1b34bc5cf35aa4b4/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:50 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_e1c7bf2add37671886c54c55f98f9fb7
>Result: meta-data=/dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_e1c7bf2add37671886c54c55f98f9fb7 isize=512    agcount=8, agsize=32768 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=262144, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:54:50 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: chown :2005 /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_dba8827a8c1b642d1b34bc5cf35aa4b4/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:50 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: awk "BEGIN {print \"/dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_e1c7bf2add37671886c54c55f98f9fb7 /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_e1c7bf2add37671886c54c55f98f9fb7 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:50 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: chmod 2775 /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_dba8827a8c1b642d1b34bc5cf35aa4b4/brick
>Result: 
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 133.244Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 127Âµs
>[kubeexec] DEBUG 2018/06/08 08:54:50 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_3a4297677881963e3f80124971d50eea/tp_b3918aae89c9aab7c5cfa3496f95936a --virtualsize 2097152K --name brick_b3918aae89c9aab7c5cfa3496f95936a
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_b3918aae89c9aab7c5cfa3496f95936a" created.
>[kubeexec] DEBUG 2018/06/08 08:54:50 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_e1c7bf2add37671886c54c55f98f9fb7 /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_e1c7bf2add37671886c54c55f98f9fb7
>Result: 
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 157.908Âµs
>[kubeexec] DEBUG 2018/06/08 08:54:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir -p /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_01b4fda3de4a6a9df5de0de9826ad1aa
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_e1c7bf2add37671886c54c55f98f9fb7/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_b3918aae89c9aab7c5cfa3496f95936a
>Result: meta-data=/dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_b3918aae89c9aab7c5cfa3496f95936a isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:54:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: awk "BEGIN {print \"/dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_b3918aae89c9aab7c5cfa3496f95936a /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_b3918aae89c9aab7c5cfa3496f95936a xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_b3918aae89c9aab7c5cfa3496f95936a /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_b3918aae89c9aab7c5cfa3496f95936a
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_b3918aae89c9aab7c5cfa3496f95936a/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_b861af9cc67f5bf8d5a8f9e38f56b14e/tp_01b4fda3de4a6a9df5de0de9826ad1aa --virtualsize 2097152K --name brick_01b4fda3de4a6a9df5de0de9826ad1aa
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_01b4fda3de4a6a9df5de0de9826ad1aa" created.
>[kubeexec] DEBUG 2018/06/08 08:54:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir -p /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_e631f0bf543f6c06867077cd16aad9e2
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: chown :2005 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_b3918aae89c9aab7c5cfa3496f95936a/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_01b4fda3de4a6a9df5de0de9826ad1aa
>Result: meta-data=/dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_01b4fda3de4a6a9df5de0de9826ad1aa isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:54:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: chmod 2775 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_b3918aae89c9aab7c5cfa3496f95936a/brick
>Result: 
>[cmdexec] INFO 2018/06/08 08:54:51 Creating volume vol_5d0dfb0ebb846fcd225c890ec9cdb885 replica 3
>[kubeexec] DEBUG 2018/06/08 08:54:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: awk "BEGIN {print \"/dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_01b4fda3de4a6a9df5de0de9826ad1aa /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_01b4fda3de4a6a9df5de0de9826ad1aa xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_01b4fda3de4a6a9df5de0de9826ad1aa /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_01b4fda3de4a6a9df5de0de9826ad1aa
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvcreate --autobackup=n --poolmetadatasize 8192K --chunksize 256K --size 1048576K --thin vg_9394bc70699b006c5460c9f654cf345f/tp_e631f0bf543f6c06867077cd16aad9e2 --virtualsize 1048576K --name brick_e631f0bf543f6c06867077cd16aad9e2
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_e631f0bf543f6c06867077cd16aad9e2" created.
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 106.266Âµs
>[kubeexec] DEBUG 2018/06/08 08:54:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir -p /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_9f2b3abdf7a755d6b95c412c440a955f
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_01b4fda3de4a6a9df5de0de9826ad1aa/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_e631f0bf543f6c06867077cd16aad9e2
>Result: meta-data=/dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_e631f0bf543f6c06867077cd16aad9e2 isize=512    agcount=8, agsize=32768 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=262144, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 224.571Âµs
>[kubeexec] DEBUG 2018/06/08 08:54:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: chown :2006 /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_01b4fda3de4a6a9df5de0de9826ad1aa/brick
>Result: 
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 130.339Âµs
>[kubeexec] DEBUG 2018/06/08 08:54:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: awk "BEGIN {print \"/dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_e631f0bf543f6c06867077cd16aad9e2 /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_e631f0bf543f6c06867077cd16aad9e2 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: chmod 2775 /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_01b4fda3de4a6a9df5de0de9826ad1aa/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_e631f0bf543f6c06867077cd16aad9e2 /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_e631f0bf543f6c06867077cd16aad9e2
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_66a9af9f6bac95f9d8d556a2f14c29d3/tp_9f2b3abdf7a755d6b95c412c440a955f --virtualsize 2097152K --name brick_9f2b3abdf7a755d6b95c412c440a955f
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_9f2b3abdf7a755d6b95c412c440a955f" created.
>[kubeexec] DEBUG 2018/06/08 08:54:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir -p /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_f399afe93703292ed5ac22602d678d6f
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_e631f0bf543f6c06867077cd16aad9e2/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_9f2b3abdf7a755d6b95c412c440a955f
>Result: meta-data=/dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_9f2b3abdf7a755d6b95c412c440a955f isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:54:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: awk "BEGIN {print \"/dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_9f2b3abdf7a755d6b95c412c440a955f /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_9f2b3abdf7a755d6b95c412c440a955f xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_9f2b3abdf7a755d6b95c412c440a955f /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_9f2b3abdf7a755d6b95c412c440a955f
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_b861af9cc67f5bf8d5a8f9e38f56b14e/tp_f399afe93703292ed5ac22602d678d6f --virtualsize 2097152K --name brick_f399afe93703292ed5ac22602d678d6f
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_f399afe93703292ed5ac22602d678d6f" created.
>[kubeexec] DEBUG 2018/06/08 08:54:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_9f2b3abdf7a755d6b95c412c440a955f/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_f399afe93703292ed5ac22602d678d6f
>Result: meta-data=/dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_f399afe93703292ed5ac22602d678d6f isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:54:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: chown :2006 /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_9f2b3abdf7a755d6b95c412c440a955f/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script snapshot list vol_9c85de66c12db0a72b4d16fe888ff74d --xml
>Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
><cliOutput>
>  <opRet>-1</opRet>
>  <opErrno>30806</opErrno>
>  <opErrstr>Volume (vol_9c85de66c12db0a72b4d16fe888ff74d) does not exist</opErrstr>
></cliOutput>
>[kubeexec] DEBUG 2018/06/08 08:54:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: awk "BEGIN {print \"/dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_f399afe93703292ed5ac22602d678d6f /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_f399afe93703292ed5ac22602d678d6f xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: chmod 2775 /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_9f2b3abdf7a755d6b95c412c440a955f/brick
>Result: 
>[cmdexec] INFO 2018/06/08 08:54:52 Creating volume vol_43194b98d83e8b61b376ccc54f79333a replica 3
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 119.774Âµs
>[kubeexec] DEBUG 2018/06/08 08:54:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_f399afe93703292ed5ac22602d678d6f /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_f399afe93703292ed5ac22602d678d6f
>Result: 
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 100.228Âµs
>[kubeexec] DEBUG 2018/06/08 08:54:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_f399afe93703292ed5ac22602d678d6f/brick
>Result: 
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 84.819Âµs
>[kubeexec] DEBUG 2018/06/08 08:54:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: chown :2007 /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_f399afe93703292ed5ac22602d678d6f/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_4f4d753d298c99eac492c32006c74484
>
>Result:   vg_96f1667f2f1ced2c5ef94772922be93b/tp_4f4d753d298c99eac492c32006c74484
>[kubeexec] DEBUG 2018/06/08 08:54:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: chmod 2775 /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_f399afe93703292ed5ac22602d678d6f/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir -p /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_e2e58ed3bab4af0c07b035a7306264ab
>Result: 
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 103.718Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 96.031Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 115.242Âµs
>[kubeexec] DEBUG 2018/06/08 08:54:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_2ed96343627983cf9667b7cee4052d17
>
>Result:   vg_9394bc70699b006c5460c9f654cf345f/tp_2ed96343627983cf9667b7cee4052d17
>[kubeexec] DEBUG 2018/06/08 08:54:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_66a9af9f6bac95f9d8d556a2f14c29d3/tp_e2e58ed3bab4af0c07b035a7306264ab --virtualsize 2097152K --name brick_e2e58ed3bab4af0c07b035a7306264ab
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_e2e58ed3bab4af0c07b035a7306264ab" created.
>[kubeexec] DEBUG 2018/06/08 08:54:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir -p /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_4f44e68452214ab813c4616ad12e2ee2
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_e2e58ed3bab4af0c07b035a7306264ab
>Result: meta-data=/dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_e2e58ed3bab4af0c07b035a7306264ab isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:54:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: awk "BEGIN {print \"/dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_e2e58ed3bab4af0c07b035a7306264ab /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_e2e58ed3bab4af0c07b035a7306264ab xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_e2e58ed3bab4af0c07b035a7306264ab /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_e2e58ed3bab4af0c07b035a7306264ab
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_e2e58ed3bab4af0c07b035a7306264ab/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvcreate --autobackup=n --poolmetadatasize 8192K --chunksize 256K --size 1048576K --thin vg_d389f0278a774bd7443a09af960961d8/tp_4f44e68452214ab813c4616ad12e2ee2 --virtualsize 1048576K --name brick_4f44e68452214ab813c4616ad12e2ee2
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_4f44e68452214ab813c4616ad12e2ee2" created.
>[kubeexec] DEBUG 2018/06/08 08:54:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: chown :2007 /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_e2e58ed3bab4af0c07b035a7306264ab/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_4f44e68452214ab813c4616ad12e2ee2
>Result: meta-data=/dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_4f44e68452214ab813c4616ad12e2ee2 isize=512    agcount=8, agsize=32768 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=262144, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:54:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: chmod 2775 /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_e2e58ed3bab4af0c07b035a7306264ab/brick
>Result: 
>[cmdexec] INFO 2018/06/08 08:54:54 Creating volume vol_7d5a429f821efc8e8fe3f29569732b86 replica 3
>[kubeexec] DEBUG 2018/06/08 08:54:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: awk "BEGIN {print \"/dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_4f44e68452214ab813c4616ad12e2ee2 /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_4f44e68452214ab813c4616ad12e2ee2 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 125.911Âµs
>[kubeexec] DEBUG 2018/06/08 08:54:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_4f44e68452214ab813c4616ad12e2ee2 /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_4f44e68452214ab813c4616ad12e2ee2
>Result: 
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 106.226Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 270.687Âµs
>[kubeexec] DEBUG 2018/06/08 08:54:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_4f44e68452214ab813c4616ad12e2ee2/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir -p /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_d105708cfed6c89ba77c7f9738020bf4
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:55 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script volume create vol_b8f78f128f20b61ef032ce9ee5b6481c replica 3 10.70.47.76:/var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_e3ebda9db9a085114fe9732ab2df4869/brick 10.70.46.187:/var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_c4423da2a93ab3c5917479262cf1d93a/brick 10.70.46.122:/var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_91d54591c1b07bc3a93d925cfbbd0f23/brick
>Result: volume create: vol_b8f78f128f20b61ef032ce9ee5b6481c: success: please start the volume to access data
>[kubeexec] DEBUG 2018/06/08 08:54:55 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvcreate --autobackup=n --poolmetadatasize 8192K --chunksize 256K --size 1048576K --thin vg_3a4297677881963e3f80124971d50eea/tp_d105708cfed6c89ba77c7f9738020bf4 --virtualsize 1048576K --name brick_d105708cfed6c89ba77c7f9738020bf4
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_d105708cfed6c89ba77c7f9738020bf4" created.
>[kubeexec] DEBUG 2018/06/08 08:54:55 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir -p /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_c36c6529173a51e7b9ae7a98545d98a2
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:55 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_d105708cfed6c89ba77c7f9738020bf4
>Result: meta-data=/dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_d105708cfed6c89ba77c7f9738020bf4 isize=512    agcount=8, agsize=32768 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=262144, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:54:55 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: awk "BEGIN {print \"/dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_d105708cfed6c89ba77c7f9738020bf4 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_d105708cfed6c89ba77c7f9738020bf4 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:55 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_d105708cfed6c89ba77c7f9738020bf4 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_d105708cfed6c89ba77c7f9738020bf4
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:55 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_d105708cfed6c89ba77c7f9738020bf4/brick
>Result: 
>[cmdexec] INFO 2018/06/08 08:54:55 Creating volume vol_aee5088d2304cb95535752ef85f9f392 replica 3
>[kubeexec] DEBUG 2018/06/08 08:54:55 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvcreate --autobackup=n --poolmetadatasize 8192K --chunksize 256K --size 1048576K --thin vg_d389f0278a774bd7443a09af960961d8/tp_c36c6529173a51e7b9ae7a98545d98a2 --virtualsize 1048576K --name brick_c36c6529173a51e7b9ae7a98545d98a2
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_c36c6529173a51e7b9ae7a98545d98a2" created.
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 128.193Âµs
>[kubeexec] DEBUG 2018/06/08 08:54:55 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir -p /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_fcac0ae8bd9895d4780d78b18d3d6c38
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:55 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_c36c6529173a51e7b9ae7a98545d98a2
>Result: meta-data=/dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_c36c6529173a51e7b9ae7a98545d98a2 isize=512    agcount=8, agsize=32768 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=262144, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 120.909Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 118.045Âµs
>[kubeexec] DEBUG 2018/06/08 08:54:55 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: awk "BEGIN {print \"/dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_c36c6529173a51e7b9ae7a98545d98a2 /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_c36c6529173a51e7b9ae7a98545d98a2 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:56 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_c36c6529173a51e7b9ae7a98545d98a2 /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_c36c6529173a51e7b9ae7a98545d98a2
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:56 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_c36c6529173a51e7b9ae7a98545d98a2/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:56 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvcreate --autobackup=n --poolmetadatasize 8192K --chunksize 256K --size 1048576K --thin vg_3a4297677881963e3f80124971d50eea/tp_fcac0ae8bd9895d4780d78b18d3d6c38 --virtualsize 1048576K --name brick_fcac0ae8bd9895d4780d78b18d3d6c38
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_fcac0ae8bd9895d4780d78b18d3d6c38" created.
>[kubeexec] DEBUG 2018/06/08 08:54:56 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir -p /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_1f46651d33f6ef49271d6d5382a7bc9c
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:56 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_fcac0ae8bd9895d4780d78b18d3d6c38
>Result: meta-data=/dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_fcac0ae8bd9895d4780d78b18d3d6c38 isize=512    agcount=8, agsize=32768 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=262144, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:54:56 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: awk "BEGIN {print \"/dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_fcac0ae8bd9895d4780d78b18d3d6c38 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_fcac0ae8bd9895d4780d78b18d3d6c38 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:56 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_fcac0ae8bd9895d4780d78b18d3d6c38 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_fcac0ae8bd9895d4780d78b18d3d6c38
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:56 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvcreate --autobackup=n --poolmetadatasize 8192K --chunksize 256K --size 1048576K --thin vg_b861af9cc67f5bf8d5a8f9e38f56b14e/tp_1f46651d33f6ef49271d6d5382a7bc9c --virtualsize 1048576K --name brick_1f46651d33f6ef49271d6d5382a7bc9c
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_1f46651d33f6ef49271d6d5382a7bc9c" created.
>[kubeexec] DEBUG 2018/06/08 08:54:56 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_fcac0ae8bd9895d4780d78b18d3d6c38/brick
>Result: 
>[cmdexec] INFO 2018/06/08 08:54:56 Creating volume vol_c1dce5388e8c136a89e4a25e4cc97821 replica 3
>[kubeexec] DEBUG 2018/06/08 08:54:56 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_1f46651d33f6ef49271d6d5382a7bc9c
>Result: meta-data=/dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_1f46651d33f6ef49271d6d5382a7bc9c isize=512    agcount=8, agsize=32768 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=262144, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:54:56 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir -p /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_f26cbb3df4880b3fe991ee4e44697c2c
>Result: 
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 106.861Âµs
>[kubeexec] DEBUG 2018/06/08 08:54:56 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: awk "BEGIN {print \"/dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_1f46651d33f6ef49271d6d5382a7bc9c /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_1f46651d33f6ef49271d6d5382a7bc9c xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 114.744Âµs
>[kubeexec] DEBUG 2018/06/08 08:54:56 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_1f46651d33f6ef49271d6d5382a7bc9c /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_1f46651d33f6ef49271d6d5382a7bc9c
>Result: 
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 127.526Âµs
>[kubeexec] DEBUG 2018/06/08 08:54:57 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_1f46651d33f6ef49271d6d5382a7bc9c/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:57 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvcreate --autobackup=n --poolmetadatasize 8192K --chunksize 256K --size 1048576K --thin vg_3a4297677881963e3f80124971d50eea/tp_f26cbb3df4880b3fe991ee4e44697c2c --virtualsize 1048576K --name brick_f26cbb3df4880b3fe991ee4e44697c2c
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_f26cbb3df4880b3fe991ee4e44697c2c" created.
>[kubeexec] DEBUG 2018/06/08 08:54:57 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir -p /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_04daa5c9d25bc1a3074533508d73b587
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:57 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_f26cbb3df4880b3fe991ee4e44697c2c
>Result: meta-data=/dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_f26cbb3df4880b3fe991ee4e44697c2c isize=512    agcount=8, agsize=32768 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=262144, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:54:57 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: awk "BEGIN {print \"/dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_f26cbb3df4880b3fe991ee4e44697c2c /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_f26cbb3df4880b3fe991ee4e44697c2c xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:57 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_f26cbb3df4880b3fe991ee4e44697c2c /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_f26cbb3df4880b3fe991ee4e44697c2c
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:57 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvcreate --autobackup=n --poolmetadatasize 8192K --chunksize 256K --size 1048576K --thin vg_d389f0278a774bd7443a09af960961d8/tp_04daa5c9d25bc1a3074533508d73b587 --virtualsize 1048576K --name brick_04daa5c9d25bc1a3074533508d73b587
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_04daa5c9d25bc1a3074533508d73b587" created.
>[kubeexec] DEBUG 2018/06/08 08:54:57 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_f26cbb3df4880b3fe991ee4e44697c2c/brick
>Result: 
>[cmdexec] INFO 2018/06/08 08:54:57 Creating volume vol_8757685f765bbd74556cbf75086c88f6 replica 3
>[kubeexec] DEBUG 2018/06/08 08:54:57 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_04daa5c9d25bc1a3074533508d73b587
>Result: meta-data=/dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_04daa5c9d25bc1a3074533508d73b587 isize=512    agcount=8, agsize=32768 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=262144, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:54:57 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir -p /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_61815093958a51df17cf62e0a12a5451
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:57 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: awk "BEGIN {print \"/dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_04daa5c9d25bc1a3074533508d73b587 /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_04daa5c9d25bc1a3074533508d73b587 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:57 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_04daa5c9d25bc1a3074533508d73b587 /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_04daa5c9d25bc1a3074533508d73b587
>Result: 
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 111.955Âµs
>[kubeexec] DEBUG 2018/06/08 08:54:57 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_04daa5c9d25bc1a3074533508d73b587/brick
>Result: 
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 170.282Âµs
>[kubeexec] DEBUG 2018/06/08 08:54:57 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvcreate --autobackup=n --poolmetadatasize 8192K --chunksize 256K --size 1048576K --thin vg_66a9af9f6bac95f9d8d556a2f14c29d3/tp_61815093958a51df17cf62e0a12a5451 --virtualsize 1048576K --name brick_61815093958a51df17cf62e0a12a5451
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_61815093958a51df17cf62e0a12a5451" created.
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 112.72Âµs
>[kubeexec] DEBUG 2018/06/08 08:54:58 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_61815093958a51df17cf62e0a12a5451
>Result: meta-data=/dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_61815093958a51df17cf62e0a12a5451 isize=512    agcount=8, agsize=32768 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=262144, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:54:58 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: awk "BEGIN {print \"/dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_61815093958a51df17cf62e0a12a5451 /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_61815093958a51df17cf62e0a12a5451 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:58 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_61815093958a51df17cf62e0a12a5451 /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_61815093958a51df17cf62e0a12a5451
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:54:58 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_61815093958a51df17cf62e0a12a5451/brick
>Result: 
>[cmdexec] INFO 2018/06/08 08:54:58 Creating volume vol_a8678c97e2708cf6e00aea160a4d46a0 replica 3
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 131.53Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 145.019Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 95.504Âµs
>[kubeexec] DEBUG 2018/06/08 08:54:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script volume start vol_b8f78f128f20b61ef032ce9ee5b6481c
>Result: volume start: vol_b8f78f128f20b61ef032ce9ee5b6481c: success
>[heketi] INFO 2018/06/08 08:54:59 Create Volume succeeded
>[asynchttp] INFO 2018/06/08 08:54:59 asynchttp.go:292: Completed job 086aebe6792ba08c072791f3d9745777 in 36.641172345s
>[kubeexec] DEBUG 2018/06/08 08:54:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: gluster --mode=script volume delete vol_ad1e5849e9566f1bcaa09cfb9c0b96ef
>Result: volume delete: vol_ad1e5849e9566f1bcaa09cfb9c0b96ef: success
>[heketi] INFO 2018/06/08 08:54:59 Deleting brick d9145089dd3be60b9df2a82315900670
>[heketi] INFO 2018/06/08 08:54:59 Deleting brick 14353da20ffb37550480dff24b915066
>[heketi] INFO 2018/06/08 08:54:59 Deleting brick e416f1b7fd62ee9320cfd9d57705d34c
>[kubeexec] DEBUG 2018/06/08 08:54:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script volume create vol_07ef5105131fa51c35a9007ee213ea7a replica 3 10.70.46.122:/var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_063b0d12615d3375614df036642dfa39/brick 10.70.47.76:/var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_4dc091ed7622300b60597d7a49fae798/brick 10.70.46.187:/var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_6bfb8b3bea31f51d5be26acc7e81ce9a/brick
>Result: volume create: vol_07ef5105131fa51c35a9007ee213ea7a: success: please start the volume to access data
>[kubeexec] DEBUG 2018/06/08 08:54:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script volume create vol_89ebb1e7eed2ff557488996a1657e75e replica 3 10.70.47.76:/var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_24c8f78999c94f4eacb087ac1fc563d1/brick 10.70.46.187:/var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_6c8f55013bee289cd516ef97746721ff/brick 10.70.46.122:/var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_c77746b3040810cf2ae10fc585c67fd5/brick
>Result: volume create: vol_89ebb1e7eed2ff557488996a1657e75e: success: please start the volume to access data
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 144.063Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 102.058Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 104.34Âµs
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 107.944Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 114.488Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 116.707Âµs
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 118.675Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 116.963Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 117.116Âµs
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 114.838Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 179.066Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 162.098Âµs
>[kubeexec] DEBUG 2018/06/08 08:55:03 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: gluster --mode=script volume create vol_a6be87754541710c38b420381c76fb8c replica 3 10.70.46.187:/var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_cb9dab914105ca3d691abed1d53d7df9/brick 10.70.46.122:/var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_47b16a0d2bab878899e523ab33a0c258/brick 10.70.47.76:/var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_cd0ffc6bd88bdb1d969dceaafda517d8/brick
>Result: volume create: vol_a6be87754541710c38b420381c76fb8c: success: please start the volume to access data
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 99.549Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 125.637Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 131.681Âµs
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 147.356Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 130.643Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 110.062Âµs
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 167.903Âµs
>[negroni] Started DELETE /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[heketi] ERROR 2018/06/08 08:55:05 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:399: Pending brick 214eb9006f9103530a1d0310d5f5dcfc can not be deleted
>[negroni] Completed 500 Internal Server Error in 4.920195ms
>[heketi] ERROR 2018/06/08 08:55:05 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:1167: Delete Volume Build Failed: The target exists, contains other items, or is in use.
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 186.835Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 103.4Âµs
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 131.235Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 145.098Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 148.187Âµs
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 118.561Âµs
>[kubeexec] DEBUG 2018/06/08 08:55:07 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script volume start vol_07ef5105131fa51c35a9007ee213ea7a
>Result: volume start: vol_07ef5105131fa51c35a9007ee213ea7a: success
>[heketi] INFO 2018/06/08 08:55:07 Create Volume succeeded
>[asynchttp] INFO 2018/06/08 08:55:07 asynchttp.go:292: Completed job 252b1a3974a7d2a66ea36f37f2c82074 in 59.809013655s
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 120.422Âµs
>[kubeexec] DEBUG 2018/06/08 08:55:07 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: umount /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_cc2483d7b49bc029b5200a024cac7535
>Result: 
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 141.784Âµs
>[kubeexec] DEBUG 2018/06/08 08:55:08 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvremove --autobackup=n -f   vg_66a9af9f6bac95f9d8d556a2f14c29d3/tp_214eb9006f9103530a1d0310d5f5dcfc
>
>Result:   Logical volume "tp_214eb9006f9103530a1d0310d5f5dcfc" successfully removed
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 175.178Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 116.33Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 166.033Âµs
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 200.009Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 120.013Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 127.211Âµs
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 177.995Âµs
>[kubeexec] DEBUG 2018/06/08 08:55:10 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script volume start vol_89ebb1e7eed2ff557488996a1657e75e
>Result: volume start: vol_89ebb1e7eed2ff557488996a1657e75e: success
>[kubeexec] DEBUG 2018/06/08 08:55:10 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: gluster --mode=script volume start vol_a6be87754541710c38b420381c76fb8c
>Result: volume start: vol_a6be87754541710c38b420381c76fb8c: success
>[kubeexec] ERROR 2018/06/08 08:55:10 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume delete vol_15e0122e942fc41f80666a3714670682] on glusterfs-storage-pg4xc: Err[command terminated with exit code 1]: Stdout []: Stderr [volume delete: vol_15e0122e942fc41f80666a3714670682: failed: Volume vol_15e0122e942fc41f80666a3714670682 does not exist
>]
>[cmdexec] ERROR 2018/06/08 08:55:10 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:153: Unable to delete volume vol_15e0122e942fc41f80666a3714670682: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_15e0122e942fc41f80666a3714670682: failed: Volume vol_15e0122e942fc41f80666a3714670682 does not exist
>[heketi] INFO 2018/06/08 08:55:10 Deleting brick c805e58953c8aa4c8e4d7298563713f7
>[heketi] INFO 2018/06/08 08:55:10 Deleting brick 30ba742d27672a5289fe7ff6bd5ef3ce
>[heketi] INFO 2018/06/08 08:55:10 Deleting brick 2aca658dfb3ac9ba0fbf538dad4caa3b
>[heketi] INFO 2018/06/08 08:55:10 Create Volume succeeded
>[asynchttp] INFO 2018/06/08 08:55:10 asynchttp.go:292: Completed job 7e56f206e49c37c88b54444c74b73010 in 48.082857251s
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 104.761Âµs
>[heketi] INFO 2018/06/08 08:55:10 Create Volume succeeded
>[asynchttp] INFO 2018/06/08 08:55:10 asynchttp.go:292: Completed job d5734675939ff78001cf479cb6e9b97c in 1m2.74765779s
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 146.997Âµs
>[kubeexec] DEBUG 2018/06/08 08:55:11 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_f05028077b974ebc1f6621aee2184169
>
>Result:   vg_d389f0278a774bd7443a09af960961d8/tp_f05028077b974ebc1f6621aee2184169
>[kubeexec] DEBUG 2018/06/08 08:55:11 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_1bb49691eccf328025855a91ee8cbc66
>
>Result:   vg_b861af9cc67f5bf8d5a8f9e38f56b14e/tp_1bb49691eccf328025855a91ee8cbc66
>[kubeexec] DEBUG 2018/06/08 08:55:11 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script volume create vol_1918777ef3ce84df17c8a114fb89f33e replica 3 10.70.46.122:/var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_cdc8c0d24f68026f06c17385d2fd0029/brick 10.70.47.76:/var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_9be3d8f4748b4c09541f8a9de90c4c34/brick 10.70.46.187:/var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_cecca27e94262fbaae81106a8cfeb0b2/brick
>Result: volume create: vol_1918777ef3ce84df17c8a114fb89f33e: success: please start the volume to access data
>[kubeexec] DEBUG 2018/06/08 08:55:11 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script volume create vol_daf13b7f607d1b280c78e909af25a215 replica 3 10.70.47.76:/var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_d8a382627ef730cbe0664d9439fd6a59/brick 10.70.46.187:/var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_0a07a24688dfe13d4e1726b8d05e24a7/brick 10.70.46.122:/var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_bd4fcfec1db80add56b57e18bd681378/brick
>Result: volume create: vol_daf13b7f607d1b280c78e909af25a215: success: please start the volume to access data
>[kubeexec] ERROR 2018/06/08 08:55:11 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume delete vol_b99532640a5201d243193159ee762ae4] on glusterfs-storage-vsh2m: Err[command terminated with exit code 1]: Stdout []: Stderr [volume delete: vol_b99532640a5201d243193159ee762ae4: failed: Volume vol_b99532640a5201d243193159ee762ae4 does not exist
>]
>[cmdexec] ERROR 2018/06/08 08:55:11 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:153: Unable to delete volume vol_b99532640a5201d243193159ee762ae4: Unable to execute command on glusterfs-storage-vsh2m: volume delete: vol_b99532640a5201d243193159ee762ae4: failed: Volume vol_b99532640a5201d243193159ee762ae4 does not exist
>[heketi] INFO 2018/06/08 08:55:11 Deleting brick dad9ada287c446b0011af8dc964060e1
>[heketi] INFO 2018/06/08 08:55:11 Deleting brick 3dbab6cd698d01ea7f00dbd81329643a
>[heketi] INFO 2018/06/08 08:55:11 Deleting brick df5559fdf15f6372e19e4e9f8bc1f129
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 118.28Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 162.966Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 116.533Âµs
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 120.348Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 132.792Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 145.415Âµs
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 115.45Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 111.765Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 98.91Âµs
>[heketi] INFO 2018/06/08 08:55:14 Starting Node Health Status refresh
>[cmdexec] INFO 2018/06/08 08:55:14 Check Glusterd service status in node dhcp46-187.lab.eng.blr.redhat.com
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 288.71Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 178.546Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 304.063Âµs
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 178.715Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 170.135Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 120.621Âµs
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 150.035Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 134.461Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 138.435Âµs
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 194.316Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 282.575Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 188.817Âµs
>[kubeexec] DEBUG 2018/06/08 08:55:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script volume start vol_1918777ef3ce84df17c8a114fb89f33e
>Result: volume start: vol_1918777ef3ce84df17c8a114fb89f33e: success
>[heketi] INFO 2018/06/08 08:55:18 Create Volume succeeded
>[asynchttp] INFO 2018/06/08 08:55:18 asynchttp.go:292: Completed job 963dc2b509a04cdd5db1a8531a35c01c in 55.946016992s
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 195.974Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 174.331Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 132.456Âµs
>[kubeexec] DEBUG 2018/06/08 08:55:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script snapshot list vol_966a92ed3e4374a5e634f7b133c49e52 --xml
>Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
><cliOutput>
>  <opRet>0</opRet>
>  <opErrno>0</opErrno>
>  <opErrstr/>
>  <snapList>
>    <count>0</count>
>  </snapList>
></cliOutput>
>[kubeexec] DEBUG 2018/06/08 08:55:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: gluster --mode=script volume create vol_dcaf25d0becadd0bb3c732c2c2ca27da replica 3 10.70.46.187:/var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_aec48b6cbce6be7256a2b86efac6aef6/brick 10.70.46.122:/var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_26379a9db79930ad00a37453b492ebac/brick 10.70.47.76:/var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_abd8495bfd886d541a449967f7bf70c0/brick
>Result: volume create: vol_dcaf25d0becadd0bb3c732c2c2ca27da: success: please start the volume to access data
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 185.178Âµs
>[kubeexec] DEBUG 2018/06/08 08:55:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script volume start vol_daf13b7f607d1b280c78e909af25a215
>Result: volume start: vol_daf13b7f607d1b280c78e909af25a215: success
>[heketi] INFO 2018/06/08 08:55:19 Create Volume succeeded
>[asynchttp] INFO 2018/06/08 08:55:19 asynchttp.go:292: Completed job 0f488f564115ab81bd476de5a18002e4 in 56.985428996s
>[kubeexec] DEBUG 2018/06/08 08:55:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script volume create vol_afef5f4a84abec3ab3e4f2a5bae2db23 replica 3 10.70.46.122:/var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_d985a824b73813069566b0a71bb73269/brick 10.70.47.76:/var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_ac27bcfa769ceba759b678d9e6b751cb/brick 10.70.46.187:/var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_dddf81e8d43f425cc26134166b21f15e/brick
>Result: volume create: vol_afef5f4a84abec3ab3e4f2a5bae2db23: success: please start the volume to access data
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 170.884Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 262.991Âµs
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 130.572Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 118.065Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 157.103Âµs
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 121.83Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 306.621Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 124.22Âµs
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 181.783Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 203.867Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 121.877Âµs
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 142.647Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 127.974Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 107.543Âµs
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 222.764Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 168.796Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 168.145Âµs
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 187.573Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 213.724Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 121.328Âµs
>[kubeexec] DEBUG 2018/06/08 08:55:26 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: gluster --mode=script volume start vol_dcaf25d0becadd0bb3c732c2c2ca27da
>Result: volume start: vol_dcaf25d0becadd0bb3c732c2c2ca27da: success
>[heketi] INFO 2018/06/08 08:55:26 Create Volume succeeded
>[asynchttp] INFO 2018/06/08 08:55:26 asynchttp.go:292: Completed job 12c7a0bf9d0844701651eb57cc4883ef in 1m3.441820767s
>[kubeexec] DEBUG 2018/06/08 08:55:26 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: gluster --mode=script snapshot list vol_a4a6e4892da299f6c5634b8a2def697e --xml
>Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
><cliOutput>
>  <opRet>0</opRet>
>  <opErrno>0</opErrno>
>  <opErrstr/>
>  <snapList>
>    <count>0</count>
>  </snapList>
></cliOutput>
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 224.548Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 255.718Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 144.213Âµs
>[kubeexec] DEBUG 2018/06/08 08:55:27 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script volume create vol_337bf2c01bf8c45eec5bab53ad5c2e46 replica 3 10.70.47.76:/var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_8d39164e3ebba9d10c8b6ed33dafabc7/brick 10.70.46.187:/var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_c740e50db586c869e27205aee2820dd7/brick 10.70.46.122:/var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_f88d16c3bc0200c12aa9db817d4d9f0a/brick
>Result: volume create: vol_337bf2c01bf8c45eec5bab53ad5c2e46: success: please start the volume to access data
>[kubeexec] DEBUG 2018/06/08 08:55:27 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script volume start vol_afef5f4a84abec3ab3e4f2a5bae2db23
>Result: volume start: vol_afef5f4a84abec3ab3e4f2a5bae2db23: success
>[heketi] INFO 2018/06/08 08:55:27 Create Volume succeeded
>[asynchttp] INFO 2018/06/08 08:55:27 asynchttp.go:292: Completed job d4f0d63988cfe91663765df353eac613 in 1m4.467266546s
>[kubeexec] DEBUG 2018/06/08 08:55:27 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: gluster --mode=script volume create vol_f73270331b95278f490fd1dfe0b010df replica 3 10.70.46.187:/var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_b35881161d7bf25bfa4af1bea4746d58/brick 10.70.46.122:/var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_a84d4290cbebd984dd11b88eb806d93b/brick 10.70.47.76:/var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_4fd98fc9e1c8a606b4b5c70bb555597e/brick
>Result: volume create: vol_f73270331b95278f490fd1dfe0b010df: success: please start the volume to access data
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 165.743Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 204.554Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 120.69Âµs
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 125.426Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 187.605Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 137.069Âµs
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 128.481Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 155.288Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 104.437Âµs
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 105.932Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 111.095Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 161.972Âµs
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 205.824Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 124.15Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 201.456Âµs
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 240.145Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 186.712Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 206.741Âµs
>[kubeexec] DEBUG 2018/06/08 08:55:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script volume start vol_337bf2c01bf8c45eec5bab53ad5c2e46
>Result: volume start: vol_337bf2c01bf8c45eec5bab53ad5c2e46: success
>[heketi] INFO 2018/06/08 08:55:33 Create Volume succeeded
>[asynchttp] INFO 2018/06/08 08:55:33 asynchttp.go:292: Completed job 44bb2a0e030ba450c0c64afe4b039c77 in 1m10.815126551s
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 196.138Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 119.357Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 134.449Âµs
>[kubeexec] DEBUG 2018/06/08 08:55:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script volume create vol_1ef58be42cf8ea7cf9298cff303e903a replica 3 10.70.46.122:/var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_2bba219914cb29c5994021b1731b7763/brick 10.70.47.76:/var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_8546c963be0cfb030d57435e94a50d61/brick 10.70.46.187:/var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_bf4ebc8a0c81383f3dc357f04e4903c1/brick
>Result: volume create: vol_1ef58be42cf8ea7cf9298cff303e903a: success: please start the volume to access data
>[kubeexec] DEBUG 2018/06/08 08:55:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: gluster --mode=script volume start vol_f73270331b95278f490fd1dfe0b010df
>Result: volume start: vol_f73270331b95278f490fd1dfe0b010df: success
>[heketi] INFO 2018/06/08 08:55:34 Create Volume succeeded
>[asynchttp] INFO 2018/06/08 08:55:34 asynchttp.go:292: Completed job addeaaa11160c5240019710f9906a045 in 1m11.857123265s
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 167.59Âµs
>[kubeexec] DEBUG 2018/06/08 08:55:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script volume create vol_b7fb8f19c77039982909b8868f7af4cc replica 3 10.70.47.76:/var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_30eb22cad2c3536d381b8d71943cffd2/brick 10.70.46.187:/var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_df153caad54b6511504d5af0cdc4e2b5/brick 10.70.46.122:/var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_54258f15bd3df0dfb8421196bdea9774/brick
>Result: volume create: vol_b7fb8f19c77039982909b8868f7af4cc: success: please start the volume to access data
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 118.983Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 155.068Âµs
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 154.238Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 158.863Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 136.192Âµs
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 169.546Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 177.893Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 165.326Âµs
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 126.061Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 161.877Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 101.66Âµs
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 199.211Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 133.166Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 271.564Âµs
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 192.558Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 126.1Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 121.431Âµs
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 245.351Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 167.236Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 163.382Âµs
>[kubeexec] DEBUG 2018/06/08 08:55:41 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script volume start vol_1ef58be42cf8ea7cf9298cff303e903a
>Result: volume start: vol_1ef58be42cf8ea7cf9298cff303e903a: success
>[heketi] INFO 2018/06/08 08:55:41 Create Volume succeeded
>[asynchttp] INFO 2018/06/08 08:55:41 asynchttp.go:292: Completed job e437810dbd1e27db60605a0a68708d27 in 1m18.177355593s
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 241.418Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 155.613Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 129.149Âµs
>[kubeexec] DEBUG 2018/06/08 08:55:42 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script volume start vol_b7fb8f19c77039982909b8868f7af4cc
>Result: volume start: vol_b7fb8f19c77039982909b8868f7af4cc: success
>[kubeexec] DEBUG 2018/06/08 08:55:42 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: gluster --mode=script volume create vol_9fb2830da79dd70d910dad8426dc236f replica 3 10.70.46.187:/var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_9963c5d95158f58c8fc2f28d81d925ea/brick 10.70.46.122:/var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_a1150dfd2cad12bad69ef9cc830da83d/brick 10.70.47.76:/var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_322eeb9ae02c4f9e338567566394d2c7/brick
>Result: volume create: vol_9fb2830da79dd70d910dad8426dc236f: success: please start the volume to access data
>[heketi] INFO 2018/06/08 08:55:42 Create Volume succeeded
>[asynchttp] INFO 2018/06/08 08:55:42 asynchttp.go:292: Completed job be6e7eb7be5d4288a84770ab21a780d4 in 1m19.133261738s
>[kubeexec] DEBUG 2018/06/08 08:55:42 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script volume create vol_1a828ff2310b778d09ffdadd755dc5ee replica 3 10.70.46.122:/var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_b2827681c0e1655380b4d5d53226bf7e/brick 10.70.47.76:/var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_6319c734fbf95bbbfb1156b24dcc43ea/brick 10.70.46.187:/var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_6167bec47d9258cf5b2a5e3a48c0d391/brick
>Result: volume create: vol_1a828ff2310b778d09ffdadd755dc5ee: success: please start the volume to access data
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 160.295Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 193.814Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 124.874Âµs
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 144.07Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 128.483Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 202.371Âµs
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 207.658Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 184.569Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 109.467Âµs
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 196.865Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 166.012Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 136.916Âµs
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 159.022Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 181.765Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 103.95Âµs
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 322.001Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 227.828Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 158.466Âµs
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 199.348Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 193.598Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 180.815Âµs
>[kubeexec] DEBUG 2018/06/08 08:55:49 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script volume create vol_e25438438fd2d50a0b07f26b4bfb338a replica 3 10.70.47.76:/var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_02dbc9628a1cd8a5f20d9fea7a615dcc/brick 10.70.46.187:/var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_e1709c1db8499b885817e5f4682c56ca/brick 10.70.46.122:/var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_a69c00298f148d2208ea7eab0903a3d2/brick
>Result: volume create: vol_e25438438fd2d50a0b07f26b4bfb338a: success: please start the volume to access data
>[kubeexec] DEBUG 2018/06/08 08:55:49 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: gluster --mode=script volume start vol_9fb2830da79dd70d910dad8426dc236f
>Result: volume start: vol_9fb2830da79dd70d910dad8426dc236f: success
>[kubeexec] DEBUG 2018/06/08 08:55:49 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script volume start vol_1a828ff2310b778d09ffdadd755dc5ee
>Result: volume start: vol_1a828ff2310b778d09ffdadd755dc5ee: success
>[heketi] INFO 2018/06/08 08:55:49 Create Volume succeeded
>[asynchttp] INFO 2018/06/08 08:55:49 asynchttp.go:292: Completed job 9a79c05867577fc3b0a49e88846f7d5d in 1m26.73862831s
>[heketi] INFO 2018/06/08 08:55:49 Create Volume succeeded
>[asynchttp] INFO 2018/06/08 08:55:49 asynchttp.go:292: Completed job 3addf5576b966f87a33743f58885b381 in 1m26.79415314s
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 108.368Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 212.585Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 140.222Âµs
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 196.628Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 258.441Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 115.811Âµs
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 125.886Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 187.265Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 132.571Âµs
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 138.895Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 172.063Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 124.661Âµs
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 184.385Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 145.928Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 165.68Âµs
>[kubeexec] DEBUG 2018/06/08 08:55:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: gluster --mode=script volume create vol_21481d8911fe8ec238d97d71c1aa5cb3 replica 3 10.70.46.187:/var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_906b174d0f419aa3d1a9affe4675674a/brick 10.70.46.122:/var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_d1e261647e36a4dd323508df5b3decff/brick 10.70.47.76:/var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_da658ca7b4961a716b886b57a67ff000/brick
>Result: volume create: vol_21481d8911fe8ec238d97d71c1aa5cb3: success: please start the volume to access data
>[kubeexec] DEBUG 2018/06/08 08:55:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script volume start vol_e25438438fd2d50a0b07f26b4bfb338a
>Result: volume start: vol_e25438438fd2d50a0b07f26b4bfb338a: success
>[kubeexec] DEBUG 2018/06/08 08:55:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script volume create vol_dc9ab13a25ccbad8262fba92766a31f9 replica 3 10.70.46.122:/var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_211fca2005300a58cf3b464b89e9deea/brick 10.70.47.76:/var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_3ca7a79bfe1ee21131ef20732eac93d7/brick 10.70.46.187:/var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_4d4160003ee228514a59c2c61932ceed/brick
>Result: volume create: vol_dc9ab13a25ccbad8262fba92766a31f9: success: please start the volume to access data
>[heketi] INFO 2018/06/08 08:55:54 Create Volume succeeded
>[asynchttp] INFO 2018/06/08 08:55:54 asynchttp.go:292: Completed job cba4a2a4640269f19e7dad88f97689c6 in 1m31.021919512s
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 222.056Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 121.84Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 166.625Âµs
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 202.462Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 172.651Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 111.836Âµs
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 181.152Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 184.553Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 136.105Âµs
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 177.905Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 216.398Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 185.884Âµs
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 148.124Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 142.769Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 116.638Âµs
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 132.05Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 225.772Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 107.552Âµs
>[kubeexec] DEBUG 2018/06/08 08:56:00 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: gluster --mode=script volume start vol_21481d8911fe8ec238d97d71c1aa5cb3
>Result: volume start: vol_21481d8911fe8ec238d97d71c1aa5cb3: success
>[heketi] INFO 2018/06/08 08:56:00 Create Volume succeeded
>[asynchttp] INFO 2018/06/08 08:56:00 asynchttp.go:292: Completed job 85bc22f147332f528f993604639f4b29 in 1m37.333423104s
>[kubeexec] DEBUG 2018/06/08 08:56:00 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mount | grep -w /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_d9145089dd3be60b9df2a82315900670 | cut -d" " -f1
>Result: /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_d9145089dd3be60b9df2a82315900670
>[kubeexec] DEBUG 2018/06/08 08:56:00 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mount | grep -w /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_c805e58953c8aa4c8e4d7298563713f7 | cut -d" " -f1
>Result: /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_c805e58953c8aa4c8e4d7298563713f7
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 211.438Âµs
>[kubeexec] DEBUG 2018/06/08 08:56:00 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: umount /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_f05028077b974ebc1f6621aee2184169
>Result: 
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 174.203Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 115.168Âµs
>[kubeexec] DEBUG 2018/06/08 08:56:01 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: umount /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_1bb49691eccf328025855a91ee8cbc66
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:56:01 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mount | grep -w /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_dad9ada287c446b0011af8dc964060e1 | cut -d" " -f1
>Result: /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_dad9ada287c446b0011af8dc964060e1
>[kubeexec] DEBUG 2018/06/08 08:56:01 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:33 UTC; 2h 32min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48fed567_6ae4_11e8_bc4b_005056a5f18a.slice/docker-a506d9aaca4b440d184df40d01e90b2040811c8090bf4c19d892081307b28c0c.scope/system.slice/glusterd.service
>           ââ  433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1228 /usr/sbin/glusterfsd -s 10.70.46.187 --volfile-id heketidbstorage.10.70.46.187.var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.187-var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.pid -S /var/run/gluster/c08898c2e3b0190c62df3a9d4a013c92.socket --brick-name /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.log --xlator-option *-posix.glusterd-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ18967 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/8fb729a0bbd51b204491c603c84acaa4.socket --xlator-option *replicate*.node-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034
>           ââ18968 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/8fb729a0bbd51b204491c603c84acaa4.socket --xlator-option *replicate*.node-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034
>
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Unit glusterd.service entered failed state.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: glusterd.service failed.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:33 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 08:56:01 Periodic health check status: node 278bd6b4e16a8e62ef15aaae22e6abc1 up=true
>[cmdexec] INFO 2018/06/08 08:56:01 Check Glusterd service status in node dhcp46-122.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 08:56:01 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script volume create vol_226838416791f3286fcacb7e5f1ff59d replica 3 10.70.47.76:/var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_0a39774f01ac7959661ff5afff2595ab/brick 10.70.46.187:/var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_592a78c302b01c3ee9538269481405e7/brick 10.70.46.122:/var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_11a59fea08d9b732822c2669ddaf54fa/brick
>Result: volume create: vol_226838416791f3286fcacb7e5f1ff59d: success: please start the volume to access data
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 135.352Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 121.526Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 87.679Âµs
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 177.595Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 140.882Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 95.23Âµs
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 145.234Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 181.125Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 185.162Âµs
>[kubeexec] DEBUG 2018/06/08 08:56:04 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script volume start vol_dc9ab13a25ccbad8262fba92766a31f9
>Result: volume start: vol_dc9ab13a25ccbad8262fba92766a31f9: success
>[kubeexec] DEBUG 2018/06/08 08:56:04 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: gluster --mode=script volume stop vol_a4a6e4892da299f6c5634b8a2def697e force
>Result: volume stop: vol_a4a6e4892da299f6c5634b8a2def697e: success
>[heketi] INFO 2018/06/08 08:56:04 Create Volume succeeded
>[asynchttp] INFO 2018/06/08 08:56:04 asynchttp.go:292: Completed job 07364f7e79063d4523983c360e41ce2a in 1m41.643590333s
>[kubeexec] DEBUG 2018/06/08 08:56:04 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_d9145089dd3be60b9df2a82315900670
>
>Result:   vg_b861af9cc67f5bf8d5a8f9e38f56b14e/tp_d9145089dd3be60b9df2a82315900670
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 119.251Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 182.882Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 121.566Âµs
>[kubeexec] DEBUG 2018/06/08 08:56:05 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_c805e58953c8aa4c8e4d7298563713f7
>
>Result:   vg_b861af9cc67f5bf8d5a8f9e38f56b14e/tp_c805e58953c8aa4c8e4d7298563713f7
>[kubeexec] DEBUG 2018/06/08 08:56:05 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: sed -i.save "/brick_f05028077b974ebc1f6621aee2184169/d" /var/lib/heketi/fstab
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:56:05 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: sed -i.save "/brick_1bb49691eccf328025855a91ee8cbc66/d" /var/lib/heketi/fstab
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:56:05 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_dad9ada287c446b0011af8dc964060e1
>
>Result:   vg_d389f0278a774bd7443a09af960961d8/tp_dad9ada287c446b0011af8dc964060e1
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 129.898Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 265.755Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 204.972Âµs
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 126.038Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 115.93Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 76.648Âµs
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 179.113Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 132.441Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 79.38Âµs
>[kubeexec] DEBUG 2018/06/08 08:56:08 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: gluster --mode=script volume delete vol_a4a6e4892da299f6c5634b8a2def697e
>Result: volume delete: vol_a4a6e4892da299f6c5634b8a2def697e: success
>[heketi] INFO 2018/06/08 08:56:08 Deleting brick 9775522b5bb908520213f17390c26d53
>[heketi] INFO 2018/06/08 08:56:08 Deleting brick b38349024b5350e969179b72b5c2af7c
>[heketi] INFO 2018/06/08 08:56:08 Deleting brick bf9bfa3d8464d1e5476516d891583f10
>[kubeexec] DEBUG 2018/06/08 08:56:08 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script volume start vol_226838416791f3286fcacb7e5f1ff59d
>Result: volume start: vol_226838416791f3286fcacb7e5f1ff59d: success
>[kubeexec] DEBUG 2018/06/08 08:56:08 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script volume create vol_d21ed39ae095af2674175a798a0cb02c replica 3 10.70.46.122:/var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_b4e462e2cc3dbccfd86e44f907dc7f00/brick 10.70.47.76:/var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_dde70bf2e86fac7df0e0427af9bf5db3/brick 10.70.46.187:/var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_2a97d96e94e0ff7c49d2b6d81dfbd8fd/brick
>Result: volume create: vol_d21ed39ae095af2674175a798a0cb02c: success: please start the volume to access data
>[heketi] INFO 2018/06/08 08:56:08 Create Volume succeeded
>[asynchttp] INFO 2018/06/08 08:56:08 asynchttp.go:292: Completed job b6fa7484737d1ad70ee715a59b9488ee in 1m45.609457708s
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 159.846Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 219.083Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 100.762Âµs
>[kubeexec] DEBUG 2018/06/08 08:56:09 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: umount /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_d9145089dd3be60b9df2a82315900670
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:56:09 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: umount /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_c805e58953c8aa4c8e4d7298563713f7
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:56:09 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvremove --autobackup=n -f /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_f05028077b974ebc1f6621aee2184169
>
>Result:   Logical volume "brick_f05028077b974ebc1f6621aee2184169" successfully removed
>[kubeexec] DEBUG 2018/06/08 08:56:09 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvremove --autobackup=n -f /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_1bb49691eccf328025855a91ee8cbc66
>
>Result:   Logical volume "brick_1bb49691eccf328025855a91ee8cbc66" successfully removed
>[kubeexec] DEBUG 2018/06/08 08:56:09 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: umount /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_dad9ada287c446b0011af8dc964060e1
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:56:09 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mount | grep -w /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_9775522b5bb908520213f17390c26d53 | cut -d" " -f1
>Result: /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_9775522b5bb908520213f17390c26d53
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 158.122Âµs
>[kubeexec] DEBUG 2018/06/08 08:56:10 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: sed -i.save "/brick_d9145089dd3be60b9df2a82315900670/d" /var/lib/heketi/fstab
>Result: 
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 131.774Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 145.159Âµs
>[kubeexec] DEBUG 2018/06/08 08:56:10 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: sed -i.save "/brick_c805e58953c8aa4c8e4d7298563713f7/d" /var/lib/heketi/fstab
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:56:10 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvs --noheadings --options=thin_count   vg_d389f0278a774bd7443a09af960961d8/tp_f05028077b974ebc1f6621aee2184169
>
>Result:        0
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 109.763Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 158.683Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 73.65Âµs
>[kubeexec] DEBUG 2018/06/08 08:56:11 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvs --noheadings --options=thin_count   vg_b861af9cc67f5bf8d5a8f9e38f56b14e/tp_1bb49691eccf328025855a91ee8cbc66
>
>Result:        0
>[kubeexec] DEBUG 2018/06/08 08:56:11 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: sed -i.save "/brick_dad9ada287c446b0011af8dc964060e1/d" /var/lib/heketi/fstab
>Result: 
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 125.148Âµs
>[kubeexec] DEBUG 2018/06/08 08:56:12 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_9775522b5bb908520213f17390c26d53
>
>Result:   vg_b861af9cc67f5bf8d5a8f9e38f56b14e/tp_9775522b5bb908520213f17390c26d53
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 100.406Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 94.522Âµs
>[kubeexec] ERROR 2018/06/08 08:56:12 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume stop vol_9c85de66c12db0a72b4d16fe888ff74d force] on glusterfs-storage-gxp7c: Err[command terminated with exit code 1]: Stdout []: Stderr [volume stop: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>]
>[cmdexec] ERROR 2018/06/08 08:56:12 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:144: Unable to stop volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume stop: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[kubeexec] DEBUG 2018/06/08 08:56:12 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script volume start vol_d21ed39ae095af2674175a798a0cb02c
>Result: volume start: vol_d21ed39ae095af2674175a798a0cb02c: success
>[heketi] INFO 2018/06/08 08:56:12 Create Volume succeeded
>[asynchttp] INFO 2018/06/08 08:56:12 asynchttp.go:292: Completed job feaab31cecfc86532b5845f95cf63913 in 1m49.028640227s
>[kubeexec] DEBUG 2018/06/08 08:56:12 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvremove --autobackup=n -f /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_d9145089dd3be60b9df2a82315900670
>
>Result:   Logical volume "brick_d9145089dd3be60b9df2a82315900670" successfully removed
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 116.039Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 124.969Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 71.95Âµs
>[kubeexec] ERROR 2018/06/08 08:56:13 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [umount /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_4f4d753d298c99eac492c32006c74484] on glusterfs-storage-gxp7c: Err[command terminated with exit code 32]: Stdout []: Stderr [umount: /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_4f4d753d298c99eac492c32006c74484: target is busy.
>        (In some cases useful info about processes that use
>         the device is found by lsof(8) or fuser(1))
>]
>[cmdexec] ERROR 2018/06/08 08:56:13 /src/github.com/heketi/heketi/executors/cmdexec/brick.go:151: <nil>
>[kubeexec] DEBUG 2018/06/08 08:56:13 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script volume create vol_5d0dfb0ebb846fcd225c890ec9cdb885 replica 3 10.70.46.122:/var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_b3918aae89c9aab7c5cfa3496f95936a/brick 10.70.47.76:/var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_c8dd6b39719a6bc75ea331fae5a92396/brick 10.70.46.187:/var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_dba8827a8c1b642d1b34bc5cf35aa4b4/brick
>Result: volume create: vol_5d0dfb0ebb846fcd225c890ec9cdb885: success: please start the volume to access data
>[kubeexec] DEBUG 2018/06/08 08:56:13 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvremove --autobackup=n -f /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_c805e58953c8aa4c8e4d7298563713f7
>
>Result:   Logical volume "brick_c805e58953c8aa4c8e4d7298563713f7" successfully removed
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 118.937Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 127.517Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 90.374Âµs
>[kubeexec] DEBUG 2018/06/08 08:56:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: umount /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_2ed96343627983cf9667b7cee4052d17
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:56:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvremove --autobackup=n -f   vg_d389f0278a774bd7443a09af960961d8/tp_f05028077b974ebc1f6621aee2184169
>
>Result:   Logical volume "tp_f05028077b974ebc1f6621aee2184169" successfully removed
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 123.566Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 115.524Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 255.379Âµs
>[kubeexec] DEBUG 2018/06/08 08:56:15 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvremove --autobackup=n -f   vg_b861af9cc67f5bf8d5a8f9e38f56b14e/tp_1bb49691eccf328025855a91ee8cbc66
>
>Result:   Logical volume "tp_1bb49691eccf328025855a91ee8cbc66" successfully removed
>[kubeexec] DEBUG 2018/06/08 08:56:15 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvremove --autobackup=n -f /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_dad9ada287c446b0011af8dc964060e1
>
>Result:   Logical volume "brick_dad9ada287c446b0011af8dc964060e1" successfully removed
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 150.41Âµs
>[kubeexec] DEBUG 2018/06/08 08:56:15 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script volume start vol_5d0dfb0ebb846fcd225c890ec9cdb885
>Result: volume start: vol_5d0dfb0ebb846fcd225c890ec9cdb885: success
>[heketi] INFO 2018/06/08 08:56:15 Create Volume succeeded
>[asynchttp] INFO 2018/06/08 08:56:15 asynchttp.go:292: Completed job 2d92a9e5a1719ab40f22667eb62b0d09 in 1m52.610923338s
>[kubeexec] DEBUG 2018/06/08 08:56:16 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: umount /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_9775522b5bb908520213f17390c26d53
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:56:16 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script volume create vol_8757685f765bbd74556cbf75086c88f6 replica 3 10.70.47.76:/var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_e1c7bf2add37671886c54c55f98f9fb7/brick 10.70.46.187:/var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_1f46651d33f6ef49271d6d5382a7bc9c/brick 10.70.46.122:/var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_f26cbb3df4880b3fe991ee4e44697c2c/brick
>Result: volume create: vol_8757685f765bbd74556cbf75086c88f6: success: please start the volume to access data
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 145.338Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 66.176Âµs
>[kubeexec] DEBUG 2018/06/08 08:56:16 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvs --noheadings --options=thin_count   vg_b861af9cc67f5bf8d5a8f9e38f56b14e/tp_d9145089dd3be60b9df2a82315900670
>
>Result:        0
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 121.243Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 85.946Âµs
>[negroni] Completed 200 OK in 243.249Âµs
>[kubeexec] DEBUG 2018/06/08 08:56:17 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvs --noheadings --options=thin_count   vg_b861af9cc67f5bf8d5a8f9e38f56b14e/tp_c805e58953c8aa4c8e4d7298563713f7
>
>Result:        0
>[kubeexec] DEBUG 2018/06/08 08:56:17 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: rmdir /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_f05028077b974ebc1f6621aee2184169
>Result: 
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 177.828Âµs
>[kubeexec] DEBUG 2018/06/08 08:56:17 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: rmdir /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_1bb49691eccf328025855a91ee8cbc66
>Result: 
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 153.239Âµs
>[negroni] Completed 200 OK in 211.145Âµs
>[kubeexec] DEBUG 2018/06/08 08:56:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvs --noheadings --options=thin_count   vg_d389f0278a774bd7443a09af960961d8/tp_dad9ada287c446b0011af8dc964060e1
>
>Result:        0
>[kubeexec] DEBUG 2018/06/08 08:56:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: sed -i.save "/brick_9775522b5bb908520213f17390c26d53/d" /var/lib/heketi/fstab
>Result: 
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 128.848Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 119.34Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 117.5Âµs
>[kubeexec] DEBUG 2018/06/08 08:56:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvremove --autobackup=n -f   vg_b861af9cc67f5bf8d5a8f9e38f56b14e/tp_d9145089dd3be60b9df2a82315900670
>
>Result:   Logical volume "tp_d9145089dd3be60b9df2a82315900670" successfully removed
>[kubeexec] DEBUG 2018/06/08 08:56:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvremove --autobackup=n -f   vg_b861af9cc67f5bf8d5a8f9e38f56b14e/tp_c805e58953c8aa4c8e4d7298563713f7
>
>Result:   Logical volume "tp_c805e58953c8aa4c8e4d7298563713f7" successfully removed
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 122.889Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 182.776Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 74.261Âµs
>[kubeexec] DEBUG 2018/06/08 08:56:20 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvremove --autobackup=n -f   vg_d389f0278a774bd7443a09af960961d8/tp_dad9ada287c446b0011af8dc964060e1
>
>Result:   Logical volume "tp_dad9ada287c446b0011af8dc964060e1" successfully removed
>[kubeexec] DEBUG 2018/06/08 08:56:20 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script volume start vol_8757685f765bbd74556cbf75086c88f6
>Result: volume start: vol_8757685f765bbd74556cbf75086c88f6: success
>[kubeexec] DEBUG 2018/06/08 08:56:20 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script volume create vol_43194b98d83e8b61b376ccc54f79333a replica 3 10.70.46.122:/var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_9f2b3abdf7a755d6b95c412c440a955f/brick 10.70.47.76:/var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_e54cabdc56fef4e5b4a11c7a72eaff3d/brick 10.70.46.187:/var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_01b4fda3de4a6a9df5de0de9826ad1aa/brick
>Result: volume create: vol_43194b98d83e8b61b376ccc54f79333a: success: please start the volume to access data
>[heketi] INFO 2018/06/08 08:56:20 Create Volume succeeded
>[asynchttp] INFO 2018/06/08 08:56:20 asynchttp.go:292: Completed job 1b65bda5683c59bd435d17b7a832a860 in 1m56.763194326s
>[kubeexec] DEBUG 2018/06/08 08:56:20 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvremove --autobackup=n -f /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_9775522b5bb908520213f17390c26d53
>
>Result:   Logical volume "brick_9775522b5bb908520213f17390c26d53" successfully removed
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 122.977Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 149.138Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 60.369Âµs
>[kubeexec] DEBUG 2018/06/08 08:56:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: rmdir /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_d9145089dd3be60b9df2a82315900670
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:56:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: rmdir /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_c805e58953c8aa4c8e4d7298563713f7
>Result: 
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 135.114Âµs
>[kubeexec] DEBUG 2018/06/08 08:56:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: rmdir /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_dad9ada287c446b0011af8dc964060e1
>Result: 
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 168.605Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 113.312Âµs
>[kubeexec] DEBUG 2018/06/08 08:56:22 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvs --noheadings --options=thin_count   vg_b861af9cc67f5bf8d5a8f9e38f56b14e/tp_9775522b5bb908520213f17390c26d53
>
>Result:        0
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 111.85Âµs
>[kubeexec] DEBUG 2018/06/08 08:56:23 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvremove --autobackup=n -f   vg_b861af9cc67f5bf8d5a8f9e38f56b14e/tp_9775522b5bb908520213f17390c26d53
>
>Result:   Logical volume "tp_9775522b5bb908520213f17390c26d53" successfully removed
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 107.161Âµs
>[negroni] Completed 200 OK in 80.291Âµs
>[kubeexec] DEBUG 2018/06/08 08:56:23 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: rmdir /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_9775522b5bb908520213f17390c26d53
>Result: 
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 170.426Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 117.107Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 308.439Âµs
>[kubeexec] DEBUG 2018/06/08 08:56:24 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script volume start vol_43194b98d83e8b61b376ccc54f79333a
>Result: volume start: vol_43194b98d83e8b61b376ccc54f79333a: success
>[kubeexec] DEBUG 2018/06/08 08:56:24 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script volume create vol_a8678c97e2708cf6e00aea160a4d46a0 replica 3 10.70.47.76:/var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_e631f0bf543f6c06867077cd16aad9e2/brick 10.70.46.187:/var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_04daa5c9d25bc1a3074533508d73b587/brick 10.70.46.122:/var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_61815093958a51df17cf62e0a12a5451/brick
>Result: volume create: vol_a8678c97e2708cf6e00aea160a4d46a0: success: please start the volume to access data
>[heketi] INFO 2018/06/08 08:56:24 Create Volume succeeded
>[asynchttp] INFO 2018/06/08 08:56:24 asynchttp.go:292: Completed job 5a4d86d3903a860d822a06fd74343b52 in 2m1.266596563s
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 223.741Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 125.118Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 61.636Âµs
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 109.914Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 92.639Âµs
>[negroni] Completed 200 OK in 182.443Âµs
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 188.832Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 186.766Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 209.532Âµs
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 170.156Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 181.688Âµs
>[negroni] Completed 200 OK in 147.872Âµs
>[kubeexec] DEBUG 2018/06/08 08:56:28 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script volume create vol_7d5a429f821efc8e8fe3f29569732b86 replica 3 10.70.46.122:/var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_e2e58ed3bab4af0c07b035a7306264ab/brick 10.70.47.76:/var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_68704fc0eb854fe2a0157b0261302792/brick 10.70.46.187:/var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_f399afe93703292ed5ac22602d678d6f/brick
>Result: volume create: vol_7d5a429f821efc8e8fe3f29569732b86: success: please start the volume to access data
>[kubeexec] DEBUG 2018/06/08 08:56:28 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script volume start vol_a8678c97e2708cf6e00aea160a4d46a0
>Result: volume start: vol_a8678c97e2708cf6e00aea160a4d46a0: success
>[heketi] INFO 2018/06/08 08:56:28 Create Volume succeeded
>[asynchttp] INFO 2018/06/08 08:56:28 asynchttp.go:292: Completed job fe707ace9f825f3ad19b3b8f28410ae4 in 2m4.467558584s
>[kubeexec] DEBUG 2018/06/08 08:56:28 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mount | grep -w /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_e416f1b7fd62ee9320cfd9d57705d34c | cut -d" " -f1
>Result: /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_e416f1b7fd62ee9320cfd9d57705d34c
>[kubeexec] DEBUG 2018/06/08 08:56:28 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mount | grep -w /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_30ba742d27672a5289fe7ff6bd5ef3ce | cut -d" " -f1
>Result: /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_30ba742d27672a5289fe7ff6bd5ef3ce
>[kubeexec] DEBUG 2018/06/08 08:56:28 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mount | grep -w /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_df5559fdf15f6372e19e4e9f8bc1f129 | cut -d" " -f1
>Result: /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_df5559fdf15f6372e19e4e9f8bc1f129
>[kubeexec] DEBUG 2018/06/08 08:56:28 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mount | grep -w /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_b38349024b5350e969179b72b5c2af7c | cut -d" " -f1
>Result: /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_b38349024b5350e969179b72b5c2af7c
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 131.578Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 155.013Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 93.949Âµs
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 261.184Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 149.706Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 93.012Âµs
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 205.352Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 147.733Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 76.161Âµs
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 200 OK in 123.801Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 122.264Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 162.439Âµs
>[kubeexec] ERROR 2018/06/08 08:56:32 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume delete vol_9c85de66c12db0a72b4d16fe888ff74d] on glusterfs-storage-gxp7c: Err[command terminated with exit code 1]: Stdout []: Stderr [volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>]
>[cmdexec] ERROR 2018/06/08 08:56:32 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:153: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[heketi] ERROR 2018/06/08 08:56:32 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:525: Unable to delete volume: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[heketi] ERROR 2018/06/08 08:56:32 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:429: Error executing delete volume: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[kubeexec] DEBUG 2018/06/08 08:56:32 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script volume start vol_7d5a429f821efc8e8fe3f29569732b86
>Result: volume start: vol_7d5a429f821efc8e8fe3f29569732b86: success
>[asynchttp] INFO 2018/06/08 08:56:32 asynchttp.go:292: Completed job 35a61a66b97e9b1b5514e032177179ad in 2m8.769856325s
>[heketi] ERROR 2018/06/08 08:56:32 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:1185: Delete Volume Failed: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[heketi] INFO 2018/06/08 08:56:32 Create Volume succeeded
>[asynchttp] INFO 2018/06/08 08:56:32 asynchttp.go:292: Completed job edf46a25868eb4962cc099193bc77737 in 2m9.019646217s
>[kubeexec] DEBUG 2018/06/08 08:56:32 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: sed -i.save "/brick_4f4d753d298c99eac492c32006c74484/d" /var/lib/heketi/fstab
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:56:32 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: sed -i.save "/brick_2ed96343627983cf9667b7cee4052d17/d" /var/lib/heketi/fstab
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:56:32 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script volume create vol_aee5088d2304cb95535752ef85f9f392 replica 3 10.70.46.122:/var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_d105708cfed6c89ba77c7f9738020bf4/brick 10.70.47.76:/var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_ad542929ce4fd8719fbe5fc44df98dbd/brick 10.70.46.187:/var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_4f44e68452214ab813c4616ad12e2ee2/brick
>Result: volume create: vol_aee5088d2304cb95535752ef85f9f392: success: please start the volume to access data
>[kubeexec] DEBUG 2018/06/08 08:56:32 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_e416f1b7fd62ee9320cfd9d57705d34c
>
>Result:   vg_96f1667f2f1ced2c5ef94772922be93b/tp_e416f1b7fd62ee9320cfd9d57705d34c
>[negroni] Started GET /queue/35a61a66b97e9b1b5514e032177179ad
>[negroni] Completed 500 Internal Server Error in 126.956Âµs
>[kubeexec] DEBUG 2018/06/08 08:56:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_30ba742d27672a5289fe7ff6bd5ef3ce
>
>Result:   vg_9394bc70699b006c5460c9f654cf345f/tp_30ba742d27672a5289fe7ff6bd5ef3ce
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 112.627Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 133.469Âµs
>[kubeexec] DEBUG 2018/06/08 08:56:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_df5559fdf15f6372e19e4e9f8bc1f129
>
>Result:   vg_96f1667f2f1ced2c5ef94772922be93b/tp_df5559fdf15f6372e19e4e9f8bc1f129
>[kubeexec] DEBUG 2018/06/08 08:56:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_b38349024b5350e969179b72b5c2af7c
>
>Result:   vg_96f1667f2f1ced2c5ef94772922be93b/tp_b38349024b5350e969179b72b5c2af7c
>[kubeexec] DEBUG 2018/06/08 08:56:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvremove --autobackup=n -f /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_2ed96343627983cf9667b7cee4052d17
>
>Result:   Logical volume "brick_2ed96343627983cf9667b7cee4052d17" successfully removed
>[kubeexec] DEBUG 2018/06/08 08:56:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: umount /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_e416f1b7fd62ee9320cfd9d57705d34c
>Result: 
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 113.34Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 82.503Âµs
>[kubeexec] DEBUG 2018/06/08 08:56:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: umount /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_30ba742d27672a5289fe7ff6bd5ef3ce
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:56:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: umount /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_df5559fdf15f6372e19e4e9f8bc1f129
>Result: 
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 127.813Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 81.702Âµs
>[kubeexec] DEBUG 2018/06/08 08:56:35 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: umount /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_b38349024b5350e969179b72b5c2af7c
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:56:35 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvs --noheadings --options=thin_count   vg_9394bc70699b006c5460c9f654cf345f/tp_2ed96343627983cf9667b7cee4052d17
>
>Result:        0
>[negroni] Started DELETE /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 202 Accepted in 20.187996ms
>[asynchttp] INFO 2018/06/08 08:56:35 asynchttp.go:288: Started job b391b58714366385bc9a2411624b4aa9
>[heketi] INFO 2018/06/08 08:56:35 Started async operation: Delete Volume
>[negroni] Started GET /queue/b391b58714366385bc9a2411624b4aa9
>[negroni] Completed 200 OK in 106.076Âµs
>[kubeexec] DEBUG 2018/06/08 08:56:35 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: sed -i.save "/brick_e416f1b7fd62ee9320cfd9d57705d34c/d" /var/lib/heketi/fstab
>Result: 
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 130.735Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 93.528Âµs
>[kubeexec] DEBUG 2018/06/08 08:56:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script volume start vol_aee5088d2304cb95535752ef85f9f392
>Result: volume start: vol_aee5088d2304cb95535752ef85f9f392: success
>[heketi] INFO 2018/06/08 08:56:36 Create Volume succeeded
>[asynchttp] INFO 2018/06/08 08:56:36 asynchttp.go:292: Completed job 01bc6bf544472a829302d04d4df8b0d1 in 2m12.644002075s
>[kubeexec] DEBUG 2018/06/08 08:56:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: sed -i.save "/brick_30ba742d27672a5289fe7ff6bd5ef3ce/d" /var/lib/heketi/fstab
>Result: 
>[negroni] Started GET /queue/b391b58714366385bc9a2411624b4aa9
>[negroni] Completed 200 OK in 148.545Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 112.699Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 77.146Âµs
>[kubeexec] DEBUG 2018/06/08 08:56:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: sed -i.save "/brick_df5559fdf15f6372e19e4e9f8bc1f129/d" /var/lib/heketi/fstab
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:56:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script volume create vol_c1dce5388e8c136a89e4a25e4cc97821 replica 3 10.70.46.122:/var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_fcac0ae8bd9895d4780d78b18d3d6c38/brick 10.70.47.76:/var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_1039ed81d7d98ea183e9c7d8d00c1b6d/brick 10.70.46.187:/var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_c36c6529173a51e7b9ae7a98545d98a2/brick
>Result: volume create: vol_c1dce5388e8c136a89e4a25e4cc97821: success: please start the volume to access data
>[kubeexec] DEBUG 2018/06/08 08:56:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: sed -i.save "/brick_b38349024b5350e969179b72b5c2af7c/d" /var/lib/heketi/fstab
>Result: 
>[negroni] Started GET /queue/b391b58714366385bc9a2411624b4aa9
>[negroni] Completed 200 OK in 143.018Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 166.725Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 296.011Âµs
>[kubeexec] DEBUG 2018/06/08 08:56:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvremove --autobackup=n -f   vg_9394bc70699b006c5460c9f654cf345f/tp_2ed96343627983cf9667b7cee4052d17
>
>Result:   Logical volume "tp_2ed96343627983cf9667b7cee4052d17" successfully removed
>[negroni] Started GET /queue/b391b58714366385bc9a2411624b4aa9
>[negroni] Completed 200 OK in 160.788Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 130.94Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 158.996Âµs
>[kubeexec] DEBUG 2018/06/08 08:56:39 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script snapshot list vol_9c85de66c12db0a72b4d16fe888ff74d --xml
>Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
><cliOutput>
>  <opRet>-1</opRet>
>  <opErrno>30806</opErrno>
>  <opErrstr>Volume (vol_9c85de66c12db0a72b4d16fe888ff74d) does not exist</opErrstr>
></cliOutput>
>[kubeexec] DEBUG 2018/06/08 08:56:39 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script volume start vol_c1dce5388e8c136a89e4a25e4cc97821
>Result: volume start: vol_c1dce5388e8c136a89e4a25e4cc97821: success
>[heketi] INFO 2018/06/08 08:56:39 Create Volume succeeded
>[asynchttp] INFO 2018/06/08 08:56:39 asynchttp.go:292: Completed job a85d28ab7c8834b5c41247d658312b8f in 2m15.922834213s
>[kubeexec] DEBUG 2018/06/08 08:56:39 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mount | grep -w /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_14353da20ffb37550480dff24b915066 | cut -d" " -f1
>Result: /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_14353da20ffb37550480dff24b915066
>[kubeexec] DEBUG 2018/06/08 08:56:39 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvremove --autobackup=n -f /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_e416f1b7fd62ee9320cfd9d57705d34c
>
>Result:   Logical volume "brick_e416f1b7fd62ee9320cfd9d57705d34c" successfully removed
>[kubeexec] DEBUG 2018/06/08 08:56:39 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: sed -i.save "/brick_cc2483d7b49bc029b5200a024cac7535/d" /var/lib/heketi/fstab
>Result: 
>[negroni] Started GET /queue/b391b58714366385bc9a2411624b4aa9
>[negroni] Completed 200 OK in 124.621Âµs
>[kubeexec] DEBUG 2018/06/08 08:56:40 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvremove --autobackup=n -f /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_30ba742d27672a5289fe7ff6bd5ef3ce
>
>Result:   Logical volume "brick_30ba742d27672a5289fe7ff6bd5ef3ce" successfully removed
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 103.952Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 57.353Âµs
>[kubeexec] DEBUG 2018/06/08 08:56:40 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: rmdir /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_214eb9006f9103530a1d0310d5f5dcfc
>Result: 
>[heketi] ERROR 2018/06/08 08:56:40 /src/github.com/heketi/heketi/apps/glusterfs/brick_create.go:73: Unable to execute command on glusterfs-storage-gxp7c: umount: /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_4f4d753d298c99eac492c32006c74484: target is busy.
>        (In some cases useful info about processes that use
>         the device is found by lsof(8) or fuser(1))
>[heketi] ERROR 2018/06/08 08:56:40 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:532: Unable to delete bricks: Unable to execute command on glusterfs-storage-gxp7c: umount: /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_4f4d753d298c99eac492c32006c74484: target is busy.
>        (In some cases useful info about processes that use
>         the device is found by lsof(8) or fuser(1))
>[heketi] ERROR 2018/06/08 08:56:40 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:429: Error executing delete volume: Unable to execute command on glusterfs-storage-gxp7c: umount: /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_4f4d753d298c99eac492c32006c74484: target is busy.
>        (In some cases useful info about processes that use
>         the device is found by lsof(8) or fuser(1))
>[asynchttp] INFO 2018/06/08 08:56:40 asynchttp.go:292: Completed job 20e932645b6c4b3853e7a46917e686e7 in 2m32.540224696s
>[heketi] ERROR 2018/06/08 08:56:40 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:1185: Delete Volume Failed: Unable to execute command on glusterfs-storage-gxp7c: umount: /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_4f4d753d298c99eac492c32006c74484: target is busy.
>        (In some cases useful info about processes that use
>         the device is found by lsof(8) or fuser(1))
>[kubeexec] DEBUG 2018/06/08 08:56:40 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvremove --autobackup=n -f /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_df5559fdf15f6372e19e4e9f8bc1f129
>
>Result:   Logical volume "brick_df5559fdf15f6372e19e4e9f8bc1f129" successfully removed
>[negroni] Started GET /queue/b391b58714366385bc9a2411624b4aa9
>[negroni] Completed 200 OK in 109.423Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 133.304Âµs
>[negroni] Completed 200 OK in 91.339Âµs
>[kubeexec] DEBUG 2018/06/08 08:56:41 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mount | grep -w /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_2aca658dfb3ac9ba0fbf538dad4caa3b | cut -d" " -f1
>Result: /dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_2aca658dfb3ac9ba0fbf538dad4caa3b
>[kubeexec] DEBUG 2018/06/08 08:56:41 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvremove --autobackup=n -f /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_b38349024b5350e969179b72b5c2af7c
>
>Result:   Logical volume "brick_b38349024b5350e969179b72b5c2af7c" successfully removed
>[negroni] Started GET /queue/b391b58714366385bc9a2411624b4aa9
>[negroni] Completed 200 OK in 157.414Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 138.64Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 106.151Âµs
>[kubeexec] DEBUG 2018/06/08 08:56:42 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mount | grep -w /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_3dbab6cd698d01ea7f00dbd81329643a | cut -d" " -f1
>Result: /dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_3dbab6cd698d01ea7f00dbd81329643a
>[kubeexec] DEBUG 2018/06/08 08:56:42 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: rmdir /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_2ed96343627983cf9667b7cee4052d17
>Result: 
>[negroni] Started GET /queue/b391b58714366385bc9a2411624b4aa9
>[negroni] Completed 200 OK in 113.897Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 99.841Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 91.832Âµs
>[negroni] Started GET /queue/b391b58714366385bc9a2411624b4aa9
>[negroni] Completed 200 OK in 128.328Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 110.681Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 132.833Âµs
>[negroni] Started GET /queue/b391b58714366385bc9a2411624b4aa9
>[negroni] Completed 200 OK in 222.242Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 117.625Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 97.063Âµs
>[negroni] Started GET /queue/b391b58714366385bc9a2411624b4aa9
>[negroni] Completed 200 OK in 194.573Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 163.955Âµs
>[negroni] Completed 200 OK in 88.158Âµs
>[kubeexec] ERROR 2018/06/08 08:56:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume stop vol_9c85de66c12db0a72b4d16fe888ff74d force] on glusterfs-storage-gxp7c: Err[command terminated with exit code 1]: Stdout []: Stderr [volume stop: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>]
>[cmdexec] ERROR 2018/06/08 08:56:46 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:144: Unable to stop volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume stop: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[kubeexec] DEBUG 2018/06/08 08:56:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script volume stop vol_966a92ed3e4374a5e634f7b133c49e52 force
>Result: volume stop: vol_966a92ed3e4374a5e634f7b133c49e52: success
>[kubeexec] DEBUG 2018/06/08 08:56:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:42 UTC; 2h 33min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4900c968_6ae4_11e8_bc4b_005056a5f18a.slice/docker-438fb5fb16a82a3e701b28fa1f1ef8623dc37ca06374c604e5781a2b2ab2e06d.scope/system.slice/glusterd.service
>           ââ  433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1106 /usr/sbin/glusterfsd -s 10.70.46.122 --volfile-id heketidbstorage.10.70.46.122.var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.122-var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.pid -S /var/run/gluster/9fa3ac3b12826912287adc88c26ac134.socket --brick-name /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_29e5b36d921cca1b0699871d9727c932/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.log --xlator-option *-posix.glusterd-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ19926 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f8994cdb09351d83e70da08261d99c64.socket --xlator-option *replicate*.node-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0
>           ââ19927 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f8994cdb09351d83e70da08261d99c64.socket --xlator-option *replicate*.node-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0
>
>Jun 08 06:23:40 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:42 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 08:56:46 Periodic health check status: node 70423cc0bcd044fe5ba8bbbd256a3e49 up=true
>[cmdexec] INFO 2018/06/08 08:56:46 Check Glusterd service status in node dhcp47-76.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 08:56:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvs --noheadings --options=thin_count   vg_96f1667f2f1ced2c5ef94772922be93b/tp_e416f1b7fd62ee9320cfd9d57705d34c
>
>Result:        0
>[kubeexec] DEBUG 2018/06/08 08:56:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mount | grep -w /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_bf9bfa3d8464d1e5476516d891583f10 | cut -d" " -f1
>Result: /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_bf9bfa3d8464d1e5476516d891583f10
>[kubeexec] DEBUG 2018/06/08 08:56:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvs --noheadings --options=thin_count   vg_9394bc70699b006c5460c9f654cf345f/tp_30ba742d27672a5289fe7ff6bd5ef3ce
>
>Result:        0
>[kubeexec] DEBUG 2018/06/08 08:56:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_14353da20ffb37550480dff24b915066
>
>Result:   vg_3a4297677881963e3f80124971d50eea/tp_14353da20ffb37550480dff24b915066
>[kubeexec] DEBUG 2018/06/08 08:56:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvs --noheadings --options=thin_count   vg_96f1667f2f1ced2c5ef94772922be93b/tp_df5559fdf15f6372e19e4e9f8bc1f129
>
>Result:        0
>[negroni] Started GET /queue/b391b58714366385bc9a2411624b4aa9
>[negroni] Completed 200 OK in 122.338Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 94.662Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 152.393Âµs
>[kubeexec] DEBUG 2018/06/08 08:56:47 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvremove --autobackup=n -f /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_cc2483d7b49bc029b5200a024cac7535
>
>Result:   Logical volume "brick_cc2483d7b49bc029b5200a024cac7535" successfully removed
>[kubeexec] DEBUG 2018/06/08 08:56:47 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvs --noheadings --options=thin_count   vg_96f1667f2f1ced2c5ef94772922be93b/tp_b38349024b5350e969179b72b5c2af7c
>
>Result:        0
>[negroni] Started GET /queue/b391b58714366385bc9a2411624b4aa9
>[negroni] Completed 200 OK in 166.266Âµs
>[kubeexec] DEBUG 2018/06/08 08:56:48 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_2aca658dfb3ac9ba0fbf538dad4caa3b
>
>Result:   vg_66a9af9f6bac95f9d8d556a2f14c29d3/tp_2aca658dfb3ac9ba0fbf538dad4caa3b
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 114.399Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 101.103Âµs
>[kubeexec] ERROR 2018/06/08 08:56:48 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume delete vol_9c85de66c12db0a72b4d16fe888ff74d] on glusterfs-storage-gxp7c: Err[command terminated with exit code 1]: Stdout []: Stderr [volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>]
>[cmdexec] ERROR 2018/06/08 08:56:48 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:153: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[heketi] ERROR 2018/06/08 08:56:48 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:525: Unable to delete volume: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[heketi] ERROR 2018/06/08 08:56:48 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:429: Error executing delete volume: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[heketi] ERROR 2018/06/08 08:56:48 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:1185: Delete Volume Failed: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[asynchttp] INFO 2018/06/08 08:56:48 asynchttp.go:292: Completed job b391b58714366385bc9a2411624b4aa9 in 12.481115401s
>[kubeexec] DEBUG 2018/06/08 08:56:48 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_3dbab6cd698d01ea7f00dbd81329643a
>
>Result:   vg_66a9af9f6bac95f9d8d556a2f14c29d3/tp_3dbab6cd698d01ea7f00dbd81329643a
>[negroni] Started GET /queue/b391b58714366385bc9a2411624b4aa9
>[negroni] Completed 500 Internal Server Error in 130.091Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 108.158Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 72.236Âµs
>[kubeexec] DEBUG 2018/06/08 08:56:49 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 07:25:20 UTC; 1h 31min ago
>  Process: 828 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 830 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5cb2299_6aec_11e8_bc4b_005056a5f18a.slice/docker-7f9223abc9a8641cea68f374f4039235515978ee2dab970ac6cbef182c535426.scope/system.slice/glusterd.service
>           ââ  830 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1197 /usr/sbin/glusterfsd -s 10.70.47.76 --volfile-id heketidbstorage.10.70.47.76.var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.76-var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.pid -S /var/run/gluster/b6976f9ddc3abc90e1dfa91c35a7c611.socket --brick-name /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_54c06cdf52c6fe1e3a3f8a9d39cde33d/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.log --xlator-option *-posix.glusterd-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ13734 /usr/sbin/glusterfsd -s 10.70.47.76 --volfile-id vol_bf31af76ef6c54e7e8f24f4d8711cb22.10.70.47.76.var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_c6935061d1c9dbb05f316e3d87080a38-brick -p /var/run/gluster/vols/vol_bf31af76ef6c54e7e8f24f4d8711cb22/10.70.47.76-var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_c6935061d1c9dbb05f316e3d87080a38-brick.pid -S /var/run/gluster/7aabe8187d4b998d498dd5d4edbd2626.socket --brick-name /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_c6935061d1c9dbb05f316e3d87080a38/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_c6935061d1c9dbb05f316e3d87080a38-brick.log --xlator-option *-posix.glusterd-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94 --brick-port 49153 --xlator-option vol_bf31af76ef6c54e7e8f24f4d8711cb22-server.listen-port=49153
>           ââ17962 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/ec468453b11caed5259f5e1abcc65b44.socket --xlator-option *replicate*.node-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94
>           ââ17963 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/ec468453b11caed5259f5e1abcc65b44.socket --xlator-option *replicate*.node-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94
>
>Jun 08 07:25:17 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 07:25:20 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Jun 08 08:53:58 dhcp47-76.lab.eng.blr.redhat.com glusterd[830]: [2018-06-08 08:53:58.338441] C [rpc-clnt.c:465:rpc_clnt_fill_request_info] 0-management: cannot lookup the saved frame corresponding to xid (103)
>[heketi] INFO 2018/06/08 08:56:49 Periodic health check status: node d942fe6c0ee5691b7cc263968f97b650 up=true
>[heketi] INFO 2018/06/08 08:56:49 Cleaned 0 nodes from health cache
>[kubeexec] DEBUG 2018/06/08 08:56:49 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script volume delete vol_966a92ed3e4374a5e634f7b133c49e52
>Result: volume delete: vol_966a92ed3e4374a5e634f7b133c49e52: success
>[heketi] INFO 2018/06/08 08:56:49 Deleting brick 00c2d7c5fda2de77f930386434235209
>[heketi] INFO 2018/06/08 08:56:49 Deleting brick 0fb75c3a027f88c53839c0f9578a1801
>[heketi] INFO 2018/06/08 08:56:49 Deleting brick 3fce0ef044cb1f17096a5bf85437e0db
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 138.792Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 132.119Âµs
>[kubeexec] DEBUG 2018/06/08 08:56:50 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvremove --autobackup=n -f   vg_96f1667f2f1ced2c5ef94772922be93b/tp_e416f1b7fd62ee9320cfd9d57705d34c
>
>Result:   Logical volume "tp_e416f1b7fd62ee9320cfd9d57705d34c" successfully removed
>[kubeexec] DEBUG 2018/06/08 08:56:50 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_bf9bfa3d8464d1e5476516d891583f10
>
>Result:   vg_3a4297677881963e3f80124971d50eea/tp_bf9bfa3d8464d1e5476516d891583f10
>[kubeexec] DEBUG 2018/06/08 08:56:50 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mount | grep -w /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_3fce0ef044cb1f17096a5bf85437e0db | cut -d" " -f1
>Result: /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_3fce0ef044cb1f17096a5bf85437e0db
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 125.379Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 93.386Âµs
>[kubeexec] DEBUG 2018/06/08 08:56:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvremove --autobackup=n -f   vg_9394bc70699b006c5460c9f654cf345f/tp_30ba742d27672a5289fe7ff6bd5ef3ce
>
>Result:   Logical volume "tp_30ba742d27672a5289fe7ff6bd5ef3ce" successfully removed
>[kubeexec] DEBUG 2018/06/08 08:56:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: umount /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_14353da20ffb37550480dff24b915066
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:56:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_3fce0ef044cb1f17096a5bf85437e0db
>
>Result:   vg_d389f0278a774bd7443a09af960961d8/tp_3fce0ef044cb1f17096a5bf85437e0db
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 100.751Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 67.529Âµs
>[kubeexec] DEBUG 2018/06/08 08:56:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvremove --autobackup=n -f   vg_96f1667f2f1ced2c5ef94772922be93b/tp_df5559fdf15f6372e19e4e9f8bc1f129
>
>Result:   Logical volume "tp_df5559fdf15f6372e19e4e9f8bc1f129" successfully removed
>[kubeexec] DEBUG 2018/06/08 08:56:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvs --noheadings --options=thin_count   vg_3a4297677881963e3f80124971d50eea/tp_cc2483d7b49bc029b5200a024cac7535
>
>Result:        0
>[kubeexec] DEBUG 2018/06/08 08:56:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: umount /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_3fce0ef044cb1f17096a5bf85437e0db
>Result: 
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 168.998Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 136.326Âµs
>[kubeexec] DEBUG 2018/06/08 08:56:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvremove --autobackup=n -f   vg_96f1667f2f1ced2c5ef94772922be93b/tp_b38349024b5350e969179b72b5c2af7c
>
>Result:   Logical volume "tp_b38349024b5350e969179b72b5c2af7c" successfully removed
>[kubeexec] DEBUG 2018/06/08 08:56:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: umount /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_2aca658dfb3ac9ba0fbf538dad4caa3b
>Result: 
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 131.67Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 73.707Âµs
>[kubeexec] DEBUG 2018/06/08 08:56:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: sed -i.save "/brick_3fce0ef044cb1f17096a5bf85437e0db/d" /var/lib/heketi/fstab
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:56:55 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mount | grep -w /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_0fb75c3a027f88c53839c0f9578a1801 | cut -d" " -f1
>Result: /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_0fb75c3a027f88c53839c0f9578a1801
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 120.73Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 103.074Âµs
>[kubeexec] DEBUG 2018/06/08 08:56:55 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: umount /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_3dbab6cd698d01ea7f00dbd81329643a
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:56:55 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvremove --autobackup=n -f /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_3fce0ef044cb1f17096a5bf85437e0db
>
>Result:   Logical volume "brick_3fce0ef044cb1f17096a5bf85437e0db" successfully removed
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 152.702Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 111.423Âµs
>[kubeexec] DEBUG 2018/06/08 08:56:56 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: rmdir /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_e416f1b7fd62ee9320cfd9d57705d34c
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:56:56 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mount | grep -w /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_00c2d7c5fda2de77f930386434235209 | cut -d" " -f1
>Result: /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_00c2d7c5fda2de77f930386434235209
>[kubeexec] DEBUG 2018/06/08 08:56:56 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvs --noheadings --options=thin_count   vg_d389f0278a774bd7443a09af960961d8/tp_3fce0ef044cb1f17096a5bf85437e0db
>
>Result:        0
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 168.876Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 99.571Âµs
>[kubeexec] DEBUG 2018/06/08 08:56:57 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: rmdir /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_30ba742d27672a5289fe7ff6bd5ef3ce
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:56:57 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: umount /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_bf9bfa3d8464d1e5476516d891583f10
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:56:57 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvremove --autobackup=n -f   vg_d389f0278a774bd7443a09af960961d8/tp_3fce0ef044cb1f17096a5bf85437e0db
>
>Result:   Logical volume "tp_3fce0ef044cb1f17096a5bf85437e0db" successfully removed
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 128.679Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 98.699Âµs
>[kubeexec] DEBUG 2018/06/08 08:56:58 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: rmdir /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_df5559fdf15f6372e19e4e9f8bc1f129
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:56:58 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: sed -i.save "/brick_14353da20ffb37550480dff24b915066/d" /var/lib/heketi/fstab
>Result: 
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 166.426Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 200.813Âµs
>[kubeexec] DEBUG 2018/06/08 08:56:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: rmdir /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_3fce0ef044cb1f17096a5bf85437e0db
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:56:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: rmdir /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_b38349024b5350e969179b72b5c2af7c
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:56:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvremove --autobackup=n -f   vg_3a4297677881963e3f80124971d50eea/tp_cc2483d7b49bc029b5200a024cac7535
>
>Result:   Logical volume "tp_cc2483d7b49bc029b5200a024cac7535" successfully removed
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 130.335Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 96.302Âµs
>[kubeexec] DEBUG 2018/06/08 08:57:00 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_0fb75c3a027f88c53839c0f9578a1801
>
>Result:   vg_9394bc70699b006c5460c9f654cf345f/tp_0fb75c3a027f88c53839c0f9578a1801
>[kubeexec] DEBUG 2018/06/08 08:57:00 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: sed -i.save "/brick_2aca658dfb3ac9ba0fbf538dad4caa3b/d" /var/lib/heketi/fstab
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:57:01 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: umount /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_0fb75c3a027f88c53839c0f9578a1801
>Result: 
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 159.139Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 136.475Âµs
>[kubeexec] DEBUG 2018/06/08 08:57:01 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: sed -i.save "/brick_3dbab6cd698d01ea7f00dbd81329643a/d" /var/lib/heketi/fstab
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:57:01 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: sed -i.save "/brick_0fb75c3a027f88c53839c0f9578a1801/d" /var/lib/heketi/fstab
>Result: 
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 125.984Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 75.996Âµs
>[kubeexec] DEBUG 2018/06/08 08:57:02 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_00c2d7c5fda2de77f930386434235209
>
>Result:   vg_3a4297677881963e3f80124971d50eea/tp_00c2d7c5fda2de77f930386434235209
>[kubeexec] DEBUG 2018/06/08 08:57:02 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvremove --autobackup=n -f /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_0fb75c3a027f88c53839c0f9578a1801
>
>Result:   Logical volume "brick_0fb75c3a027f88c53839c0f9578a1801" successfully removed
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 91.307Âµs
>[negroni] Completed 200 OK in 389.813Âµs
>[kubeexec] DEBUG 2018/06/08 08:57:03 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: sed -i.save "/brick_bf9bfa3d8464d1e5476516d891583f10/d" /var/lib/heketi/fstab
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:57:03 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvs --noheadings --options=thin_count   vg_9394bc70699b006c5460c9f654cf345f/tp_0fb75c3a027f88c53839c0f9578a1801
>
>Result:        0
>[kubeexec] DEBUG 2018/06/08 08:57:04 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvremove --autobackup=n -f /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_14353da20ffb37550480dff24b915066
>
>Result:   Logical volume "brick_14353da20ffb37550480dff24b915066" successfully removed
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 119.056Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 92.233Âµs
>[kubeexec] DEBUG 2018/06/08 08:57:04 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvremove --autobackup=n -f   vg_9394bc70699b006c5460c9f654cf345f/tp_0fb75c3a027f88c53839c0f9578a1801
>
>Result:   Logical volume "tp_0fb75c3a027f88c53839c0f9578a1801" successfully removed
>[kubeexec] DEBUG 2018/06/08 08:57:04 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: rmdir /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_cc2483d7b49bc029b5200a024cac7535
>Result: 
>[heketi] INFO 2018/06/08 08:57:04 Allocating brick set #0
>[heketi] INFO 2018/06/08 08:57:04 Creating brick bd498e4984809a277823f95d7ac56f3a
>[heketi] INFO 2018/06/08 08:57:04 Creating brick f6df34957c593ea7de0d475a1ccbded6
>[heketi] INFO 2018/06/08 08:57:04 Creating brick e9b7f209aa8449a1dc89d03f1f8aa3ed
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 112.166Âµs
>[negroni] Completed 200 OK in 359.668Âµs
>[kubeexec] DEBUG 2018/06/08 08:57:05 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: rmdir /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_0fb75c3a027f88c53839c0f9578a1801
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:57:05 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvremove --autobackup=n -f /dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_2aca658dfb3ac9ba0fbf538dad4caa3b
>
>Result:   Logical volume "brick_2aca658dfb3ac9ba0fbf538dad4caa3b" successfully removed
>[kubeexec] DEBUG 2018/06/08 08:57:05 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir -p /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_f6df34957c593ea7de0d475a1ccbded6
>Result: 
>[negroni] Started DELETE /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 202 Accepted in 17.132662ms
>[asynchttp] INFO 2018/06/08 08:57:05 asynchttp.go:288: Started job e21c37500d0d0bb1407abc0ab8b08a7e
>[heketi] INFO 2018/06/08 08:57:05 Started async operation: Delete Volume
>[negroni] Started GET /queue/e21c37500d0d0bb1407abc0ab8b08a7e
>[negroni] Completed 200 OK in 118.332Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 168.052Âµs
>[negroni] Completed 200 OK in 135.926Âµs
>[kubeexec] DEBUG 2018/06/08 08:57:06 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_b861af9cc67f5bf8d5a8f9e38f56b14e/tp_f6df34957c593ea7de0d475a1ccbded6 --virtualsize 2097152K --name brick_f6df34957c593ea7de0d475a1ccbded6
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_f6df34957c593ea7de0d475a1ccbded6" created.
>[kubeexec] DEBUG 2018/06/08 08:57:06 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir -p /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_e9b7f209aa8449a1dc89d03f1f8aa3ed
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:57:06 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_f6df34957c593ea7de0d475a1ccbded6
>Result: meta-data=/dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_f6df34957c593ea7de0d475a1ccbded6 isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:57:06 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: awk "BEGIN {print \"/dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_f6df34957c593ea7de0d475a1ccbded6 /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_f6df34957c593ea7de0d475a1ccbded6 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:57:06 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvremove --autobackup=n -f /dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_3dbab6cd698d01ea7f00dbd81329643a
>
>Result:   Logical volume "brick_3dbab6cd698d01ea7f00dbd81329643a" successfully removed
>[kubeexec] DEBUG 2018/06/08 08:57:06 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_f6df34957c593ea7de0d475a1ccbded6 /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_f6df34957c593ea7de0d475a1ccbded6
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:57:06 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_96f1667f2f1ced2c5ef94772922be93b/tp_e9b7f209aa8449a1dc89d03f1f8aa3ed --virtualsize 2097152K --name brick_e9b7f209aa8449a1dc89d03f1f8aa3ed
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_e9b7f209aa8449a1dc89d03f1f8aa3ed" created.
>[kubeexec] DEBUG 2018/06/08 08:57:06 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_f6df34957c593ea7de0d475a1ccbded6/brick
>Result: 
>[negroni] Started GET /queue/e21c37500d0d0bb1407abc0ab8b08a7e
>[negroni] Completed 200 OK in 154.503Âµs
>[kubeexec] DEBUG 2018/06/08 08:57:06 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: chown :2002 /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_f6df34957c593ea7de0d475a1ccbded6/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:57:06 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_e9b7f209aa8449a1dc89d03f1f8aa3ed
>Result: meta-data=/dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_e9b7f209aa8449a1dc89d03f1f8aa3ed isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:57:07 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: chmod 2775 /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_f6df34957c593ea7de0d475a1ccbded6/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:57:07 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: umount /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_00c2d7c5fda2de77f930386434235209
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:57:07 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: awk "BEGIN {print \"/dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_e9b7f209aa8449a1dc89d03f1f8aa3ed /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_e9b7f209aa8449a1dc89d03f1f8aa3ed xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 159.352Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 116.352Âµs
>[kubeexec] DEBUG 2018/06/08 08:57:07 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_e9b7f209aa8449a1dc89d03f1f8aa3ed /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_e9b7f209aa8449a1dc89d03f1f8aa3ed
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:57:07 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_e9b7f209aa8449a1dc89d03f1f8aa3ed/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:57:07 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: chown :2002 /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_e9b7f209aa8449a1dc89d03f1f8aa3ed/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:57:07 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvremove --autobackup=n -f /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_bf9bfa3d8464d1e5476516d891583f10
>
>Result:   Logical volume "brick_bf9bfa3d8464d1e5476516d891583f10" successfully removed
>[kubeexec] DEBUG 2018/06/08 08:57:07 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: chmod 2775 /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_e9b7f209aa8449a1dc89d03f1f8aa3ed/brick
>Result: 
>[negroni] Started GET /queue/e21c37500d0d0bb1407abc0ab8b08a7e
>[negroni] Completed 200 OK in 134.271Âµs
>[kubeexec] DEBUG 2018/06/08 08:57:08 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvs --noheadings --options=thin_count   vg_3a4297677881963e3f80124971d50eea/tp_14353da20ffb37550480dff24b915066
>
>Result:        0
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 105.398Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 58.299Âµs
>[kubeexec] DEBUG 2018/06/08 08:57:08 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script snapshot list vol_9c85de66c12db0a72b4d16fe888ff74d --xml
>Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
><cliOutput>
>  <opRet>-1</opRet>
>  <opErrno>30806</opErrno>
>  <opErrstr>Volume (vol_9c85de66c12db0a72b4d16fe888ff74d) does not exist</opErrstr>
></cliOutput>
>[kubeexec] DEBUG 2018/06/08 08:57:08 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir -p /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_bd498e4984809a277823f95d7ac56f3a
>Result: 
>[negroni] Started GET /queue/e21c37500d0d0bb1407abc0ab8b08a7e
>[negroni] Completed 200 OK in 183.018Âµs
>[kubeexec] ERROR 2018/06/08 08:57:09 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume stop vol_9c85de66c12db0a72b4d16fe888ff74d force] on glusterfs-storage-gxp7c: Err[command terminated with exit code 1]: Stdout []: Stderr [volume stop: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>]
>[cmdexec] ERROR 2018/06/08 08:57:09 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:144: Unable to stop volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume stop: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[kubeexec] DEBUG 2018/06/08 08:57:09 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_66a9af9f6bac95f9d8d556a2f14c29d3/tp_bd498e4984809a277823f95d7ac56f3a --virtualsize 2097152K --name brick_bd498e4984809a277823f95d7ac56f3a
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_bd498e4984809a277823f95d7ac56f3a" created.
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 97.834Âµs
>[negroni] Completed 200 OK in 198.583Âµs
>[kubeexec] DEBUG 2018/06/08 08:57:09 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_bd498e4984809a277823f95d7ac56f3a
>Result: meta-data=/dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_bd498e4984809a277823f95d7ac56f3a isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:57:09 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: awk "BEGIN {print \"/dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_bd498e4984809a277823f95d7ac56f3a /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_bd498e4984809a277823f95d7ac56f3a xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:57:09 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_bd498e4984809a277823f95d7ac56f3a /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_bd498e4984809a277823f95d7ac56f3a
>Result: 
>[kubeexec] ERROR 2018/06/08 08:57:09 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume delete vol_9c85de66c12db0a72b4d16fe888ff74d] on glusterfs-storage-gxp7c: Err[command terminated with exit code 1]: Stdout []: Stderr [volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>]
>[cmdexec] ERROR 2018/06/08 08:57:09 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:153: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[heketi] ERROR 2018/06/08 08:57:09 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:525: Unable to delete volume: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[heketi] ERROR 2018/06/08 08:57:09 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:429: Error executing delete volume: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[heketi] ERROR 2018/06/08 08:57:09 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:1185: Delete Volume Failed: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[asynchttp] INFO 2018/06/08 08:57:09 asynchttp.go:292: Completed job e21c37500d0d0bb1407abc0ab8b08a7e in 3.670301234s
>[kubeexec] DEBUG 2018/06/08 08:57:09 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_bd498e4984809a277823f95d7ac56f3a/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:57:09 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: chown :2002 /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_bd498e4984809a277823f95d7ac56f3a/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:57:09 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: chmod 2775 /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_bd498e4984809a277823f95d7ac56f3a/brick
>Result: 
>[cmdexec] INFO 2018/06/08 08:57:09 Creating volume vol_9f9fb17746d0aad637b132875d2744e5 replica 3
>[negroni] Started GET /queue/e21c37500d0d0bb1407abc0ab8b08a7e
>[negroni] Completed 500 Internal Server Error in 116.502Âµs
>[kubeexec] DEBUG 2018/06/08 08:57:09 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvs --noheadings --options=thin_count   vg_66a9af9f6bac95f9d8d556a2f14c29d3/tp_2aca658dfb3ac9ba0fbf538dad4caa3b
>
>Result:        0
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 141.891Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 116.01Âµs
>[kubeexec] DEBUG 2018/06/08 08:57:10 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvs --noheadings --options=thin_count   vg_66a9af9f6bac95f9d8d556a2f14c29d3/tp_3dbab6cd698d01ea7f00dbd81329643a
>
>Result:        0
>[kubeexec] DEBUG 2018/06/08 08:57:10 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: sed -i.save "/brick_00c2d7c5fda2de77f930386434235209/d" /var/lib/heketi/fstab
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:57:11 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvs --noheadings --options=thin_count   vg_3a4297677881963e3f80124971d50eea/tp_bf9bfa3d8464d1e5476516d891583f10
>
>Result:        0
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 226.095Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 223.012Âµs
>[kubeexec] DEBUG 2018/06/08 08:57:11 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvremove --autobackup=n -f   vg_3a4297677881963e3f80124971d50eea/tp_14353da20ffb37550480dff24b915066
>
>Result:   Logical volume "tp_14353da20ffb37550480dff24b915066" successfully removed
>[kubeexec] DEBUG 2018/06/08 08:57:11 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script volume create vol_9f9fb17746d0aad637b132875d2744e5 replica 3 10.70.46.122:/var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_bd498e4984809a277823f95d7ac56f3a/brick 10.70.47.76:/var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_e9b7f209aa8449a1dc89d03f1f8aa3ed/brick 10.70.46.187:/var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_f6df34957c593ea7de0d475a1ccbded6/brick
>Result: volume create: vol_9f9fb17746d0aad637b132875d2744e5: success: please start the volume to access data
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 91.057Âµs
>[negroni] Completed 200 OK in 825.018Âµs
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 132.608Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 62.554Âµs
>[heketi] INFO 2018/06/08 08:57:14 Starting Node Health Status refresh
>[cmdexec] INFO 2018/06/08 08:57:14 Check Glusterd service status in node dhcp46-187.lab.eng.blr.redhat.com
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 111.076Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 76.999Âµs
>[kubeexec] DEBUG 2018/06/08 08:57:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script volume start vol_9f9fb17746d0aad637b132875d2744e5
>Result: volume start: vol_9f9fb17746d0aad637b132875d2744e5: success
>[kubeexec] DEBUG 2018/06/08 08:57:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:33 UTC; 2h 33min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48fed567_6ae4_11e8_bc4b_005056a5f18a.slice/docker-a506d9aaca4b440d184df40d01e90b2040811c8090bf4c19d892081307b28c0c.scope/system.slice/glusterd.service
>           ââ  433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1228 /usr/sbin/glusterfsd -s 10.70.46.187 --volfile-id heketidbstorage.10.70.46.187.var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.187-var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.pid -S /var/run/gluster/c08898c2e3b0190c62df3a9d4a013c92.socket --brick-name /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.log --xlator-option *-posix.glusterd-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ20060 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/8fb729a0bbd51b204491c603c84acaa4.socket --xlator-option *replicate*.node-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034
>           ââ20061 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/8fb729a0bbd51b204491c603c84acaa4.socket --xlator-option *replicate*.node-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034
>           ââ20076 /bin/bash /var/lib/glusterd/hooks/1/start/post/S30samba-start.sh --volname=vol_9f9fb17746d0aad637b132875d2744e5 --first=no --version=1 --volume-op=start --gd-workdir=/var/lib/glusterd
>           ââ20078 /bin/bash /var/lib/glusterd/hooks/1/start/post/S30samba-start.sh --volname=vol_9f9fb17746d0aad637b132875d2744e5 --first=no --version=1 --volume-op=start --gd-workdir=/var/lib/glusterd
>           ââ20079 /bin/bash /var/lib/glusterd/hooks/1/start/post/S30samba-start.sh --volname=vol_9f9fb17746d0aad637b132875d2744e5 --first=no --version=1 --volume-op=start --gd-workdir=/var/lib/glusterd
>           ââ20080 grep user.cifs /var/lib/glusterd/vols/vol_9f9fb17746d0aad637b132875d2744e5/info
>           ââ20081 cut -d= -f2
>
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Unit glusterd.service entered failed state.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: glusterd.service failed.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:33 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 08:57:14 Periodic health check status: node 278bd6b4e16a8e62ef15aaae22e6abc1 up=true
>[cmdexec] INFO 2018/06/08 08:57:14 Check Glusterd service status in node dhcp46-122.lab.eng.blr.redhat.com
>[asynchttp] INFO 2018/06/08 08:57:14 asynchttp.go:292: Completed job d6cbe9a27af546f3c96e74f32e86cd27 in 7m21.17200539s
>[kubeexec] DEBUG 2018/06/08 08:57:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvremove --autobackup=n -f   vg_66a9af9f6bac95f9d8d556a2f14c29d3/tp_2aca658dfb3ac9ba0fbf538dad4caa3b
>
>Result:   Logical volume "tp_2aca658dfb3ac9ba0fbf538dad4caa3b" successfully removed
>[kubeexec] DEBUG 2018/06/08 08:57:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvremove --autobackup=n -f   vg_66a9af9f6bac95f9d8d556a2f14c29d3/tp_3dbab6cd698d01ea7f00dbd81329643a
>
>Result:   Logical volume "tp_3dbab6cd698d01ea7f00dbd81329643a" successfully removed
>[kubeexec] DEBUG 2018/06/08 08:57:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvremove --autobackup=n -f /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_00c2d7c5fda2de77f930386434235209
>
>Result:   Logical volume "brick_00c2d7c5fda2de77f930386434235209" successfully removed
>[kubeexec] DEBUG 2018/06/08 08:57:15 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvremove --autobackup=n -f   vg_3a4297677881963e3f80124971d50eea/tp_bf9bfa3d8464d1e5476516d891583f10
>
>Result:   Logical volume "tp_bf9bfa3d8464d1e5476516d891583f10" successfully removed
>[kubeexec] DEBUG 2018/06/08 08:57:15 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: rmdir /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_14353da20ffb37550480dff24b915066
>Result: 
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 105.119Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 92.291Âµs
>[heketi] INFO 2018/06/08 08:57:15 Allocating brick set #0
>[kubeexec] DEBUG 2018/06/08 08:57:15 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:42 UTC; 2h 33min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4900c968_6ae4_11e8_bc4b_005056a5f18a.slice/docker-438fb5fb16a82a3e701b28fa1f1ef8623dc37ca06374c604e5781a2b2ab2e06d.scope/system.slice/glusterd.service
>           ââ  433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1106 /usr/sbin/glusterfsd -s 10.70.46.122 --volfile-id heketidbstorage.10.70.46.122.var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.122-var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.pid -S /var/run/gluster/9fa3ac3b12826912287adc88c26ac134.socket --brick-name /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_29e5b36d921cca1b0699871d9727c932/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.log --xlator-option *-posix.glusterd-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ20425 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f8994cdb09351d83e70da08261d99c64.socket --xlator-option *replicate*.node-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0
>           ââ20426 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f8994cdb09351d83e70da08261d99c64.socket --xlator-option *replicate*.node-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0
>
>Jun 08 06:23:40 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:42 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 08:57:15 Periodic health check status: node 70423cc0bcd044fe5ba8bbbd256a3e49 up=true
>[cmdexec] INFO 2018/06/08 08:57:15 Check Glusterd service status in node dhcp47-76.lab.eng.blr.redhat.com
>[heketi] INFO 2018/06/08 08:57:15 Creating brick 7649722a655922fad2e75ee7af97c506
>[heketi] INFO 2018/06/08 08:57:15 Creating brick 94745514333fff7f22e8d60eff97d635
>[heketi] INFO 2018/06/08 08:57:15 Creating brick a70d481dd9566d346b4d60131fb23a9c
>[kubeexec] DEBUG 2018/06/08 08:57:15 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: rmdir /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_2aca658dfb3ac9ba0fbf538dad4caa3b
>Result: 
>[heketi] INFO 2018/06/08 08:57:15 Allocating brick set #0
>[kubeexec] DEBUG 2018/06/08 08:57:15 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 07:25:20 UTC; 1h 31min ago
>  Process: 828 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 830 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5cb2299_6aec_11e8_bc4b_005056a5f18a.slice/docker-7f9223abc9a8641cea68f374f4039235515978ee2dab970ac6cbef182c535426.scope/system.slice/glusterd.service
>           ââ  830 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1197 /usr/sbin/glusterfsd -s 10.70.47.76 --volfile-id heketidbstorage.10.70.47.76.var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.76-var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.pid -S /var/run/gluster/b6976f9ddc3abc90e1dfa91c35a7c611.socket --brick-name /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_54c06cdf52c6fe1e3a3f8a9d39cde33d/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.log --xlator-option *-posix.glusterd-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ13734 /usr/sbin/glusterfsd -s 10.70.47.76 --volfile-id vol_bf31af76ef6c54e7e8f24f4d8711cb22.10.70.47.76.var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_c6935061d1c9dbb05f316e3d87080a38-brick -p /var/run/gluster/vols/vol_bf31af76ef6c54e7e8f24f4d8711cb22/10.70.47.76-var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_c6935061d1c9dbb05f316e3d87080a38-brick.pid -S /var/run/gluster/7aabe8187d4b998d498dd5d4edbd2626.socket --brick-name /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_c6935061d1c9dbb05f316e3d87080a38/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_c6935061d1c9dbb05f316e3d87080a38-brick.log --xlator-option *-posix.glusterd-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94 --brick-port 49153 --xlator-option vol_bf31af76ef6c54e7e8f24f4d8711cb22-server.listen-port=49153
>           ââ18430 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/ec468453b11caed5259f5e1abcc65b44.socket --xlator-option *replicate*.node-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94
>           ââ18431 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/ec468453b11caed5259f5e1abcc65b44.socket --xlator-option *replicate*.node-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94
>
>Jun 08 07:25:17 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 07:25:20 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Jun 08 08:53:58 dhcp47-76.lab.eng.blr.redhat.com glusterd[830]: [2018-06-08 08:53:58.338441] C [rpc-clnt.c:465:rpc_clnt_fill_request_info] 0-management: cannot lookup the saved frame corresponding to xid (103)
>[heketi] INFO 2018/06/08 08:57:15 Periodic health check status: node d942fe6c0ee5691b7cc263968f97b650 up=true
>[heketi] INFO 2018/06/08 08:57:15 Cleaned 0 nodes from health cache
>[heketi] INFO 2018/06/08 08:57:16 Creating brick d11702f97397089b0224af9ef7a122f1
>[heketi] INFO 2018/06/08 08:57:16 Creating brick b6a2ea41f37c6396a2e85a6a41faa54f
>[heketi] INFO 2018/06/08 08:57:16 Creating brick 070c89e7be12165f271c2906ae9e74b7
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 109.701Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 74.12Âµs
>[kubeexec] DEBUG 2018/06/08 08:57:16 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir -p /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_a70d481dd9566d346b4d60131fb23a9c
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:57:16 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_b861af9cc67f5bf8d5a8f9e38f56b14e/tp_a70d481dd9566d346b4d60131fb23a9c --virtualsize 2097152K --name brick_a70d481dd9566d346b4d60131fb23a9c
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_a70d481dd9566d346b4d60131fb23a9c" created.
>[kubeexec] DEBUG 2018/06/08 08:57:16 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: rmdir /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_3dbab6cd698d01ea7f00dbd81329643a
>Result: 
>[heketi] INFO 2018/06/08 08:57:16 Allocating brick set #0
>[heketi] INFO 2018/06/08 08:57:16 Creating brick 81fe2c773a661172f7014e85c2f68184
>[heketi] INFO 2018/06/08 08:57:16 Creating brick 9049906389b8fcca23da885bdc1ce8ab
>[heketi] INFO 2018/06/08 08:57:16 Creating brick e4aaed99528f9d907b84f92e740f0da4
>[kubeexec] DEBUG 2018/06/08 08:57:16 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_a70d481dd9566d346b4d60131fb23a9c
>Result: meta-data=/dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_a70d481dd9566d346b4d60131fb23a9c isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:57:16 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: awk "BEGIN {print \"/dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_a70d481dd9566d346b4d60131fb23a9c /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_a70d481dd9566d346b4d60131fb23a9c xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:57:16 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir -p /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_94745514333fff7f22e8d60eff97d635
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:57:17 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_a70d481dd9566d346b4d60131fb23a9c /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_a70d481dd9566d346b4d60131fb23a9c
>Result: 
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 106.154Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 99.818Âµs
>[kubeexec] DEBUG 2018/06/08 08:57:17 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_a70d481dd9566d346b4d60131fb23a9c/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:57:17 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: chown :2000 /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_a70d481dd9566d346b4d60131fb23a9c/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:57:17 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_9394bc70699b006c5460c9f654cf345f/tp_94745514333fff7f22e8d60eff97d635 --virtualsize 2097152K --name brick_94745514333fff7f22e8d60eff97d635
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_94745514333fff7f22e8d60eff97d635" created.
>[kubeexec] DEBUG 2018/06/08 08:57:17 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: chmod 2775 /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_a70d481dd9566d346b4d60131fb23a9c/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:57:17 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvs --noheadings --options=thin_count   vg_3a4297677881963e3f80124971d50eea/tp_00c2d7c5fda2de77f930386434235209
>
>Result:        0
>[kubeexec] DEBUG 2018/06/08 08:57:17 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_94745514333fff7f22e8d60eff97d635
>Result: meta-data=/dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_94745514333fff7f22e8d60eff97d635 isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:57:17 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: awk "BEGIN {print \"/dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_94745514333fff7f22e8d60eff97d635 /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_94745514333fff7f22e8d60eff97d635 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:57:17 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_94745514333fff7f22e8d60eff97d635 /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_94745514333fff7f22e8d60eff97d635
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:57:17 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_94745514333fff7f22e8d60eff97d635/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:57:17 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir -p /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_b6a2ea41f37c6396a2e85a6a41faa54f
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:57:17 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: chown :2000 /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_94745514333fff7f22e8d60eff97d635/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:57:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: chmod 2775 /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_94745514333fff7f22e8d60eff97d635/brick
>Result: 
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 200 OK in 160.668Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 129.412Âµs
>[kubeexec] DEBUG 2018/06/08 08:57:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: rmdir /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_bf9bfa3d8464d1e5476516d891583f10
>Result: 
>[heketi] INFO 2018/06/08 08:57:18 Delete Volume succeeded
>[asynchttp] INFO 2018/06/08 08:57:18 asynchttp.go:292: Completed job 30a446e1b3f0ca388030dee9d6823363 in 2m42.319785934s
>[kubeexec] DEBUG 2018/06/08 08:57:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_b861af9cc67f5bf8d5a8f9e38f56b14e/tp_b6a2ea41f37c6396a2e85a6a41faa54f --virtualsize 2097152K --name brick_b6a2ea41f37c6396a2e85a6a41faa54f
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_b6a2ea41f37c6396a2e85a6a41faa54f" created.
>[kubeexec] DEBUG 2018/06/08 08:57:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_b6a2ea41f37c6396a2e85a6a41faa54f
>Result: meta-data=/dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_b6a2ea41f37c6396a2e85a6a41faa54f isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:57:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: awk "BEGIN {print \"/dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_b6a2ea41f37c6396a2e85a6a41faa54f /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_b6a2ea41f37c6396a2e85a6a41faa54f xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:57:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_b6a2ea41f37c6396a2e85a6a41faa54f /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_b6a2ea41f37c6396a2e85a6a41faa54f
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:57:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir -p /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_070c89e7be12165f271c2906ae9e74b7
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:57:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_b6a2ea41f37c6396a2e85a6a41faa54f/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:57:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir -p /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_7649722a655922fad2e75ee7af97c506
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:57:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: chown :2001 /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_b6a2ea41f37c6396a2e85a6a41faa54f/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:57:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: chmod 2775 /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_b6a2ea41f37c6396a2e85a6a41faa54f/brick
>Result: 
>[negroni] Started GET /queue/30a446e1b3f0ca388030dee9d6823363
>[negroni] Completed 204 No Content in 99.683Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 98.67Âµs
>[kubeexec] DEBUG 2018/06/08 08:57:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_9394bc70699b006c5460c9f654cf345f/tp_070c89e7be12165f271c2906ae9e74b7 --virtualsize 2097152K --name brick_070c89e7be12165f271c2906ae9e74b7
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_070c89e7be12165f271c2906ae9e74b7" created.
>[negroni] Started DELETE /volumes/a4a6e4892da299f6c5634b8a2def697e
>[negroni] Completed 404 Not Found in 13.671837ms
>[kubeexec] DEBUG 2018/06/08 08:57:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_070c89e7be12165f271c2906ae9e74b7
>Result: meta-data=/dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_070c89e7be12165f271c2906ae9e74b7 isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:57:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir -p /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_9049906389b8fcca23da885bdc1ce8ab
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:57:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_66a9af9f6bac95f9d8d556a2f14c29d3/tp_7649722a655922fad2e75ee7af97c506 --virtualsize 2097152K --name brick_7649722a655922fad2e75ee7af97c506
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_7649722a655922fad2e75ee7af97c506" created.
>[kubeexec] DEBUG 2018/06/08 08:57:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: awk "BEGIN {print \"/dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_070c89e7be12165f271c2906ae9e74b7 /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_070c89e7be12165f271c2906ae9e74b7 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:57:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_7649722a655922fad2e75ee7af97c506
>Result: meta-data=/dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_7649722a655922fad2e75ee7af97c506 isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:57:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_070c89e7be12165f271c2906ae9e74b7 /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_070c89e7be12165f271c2906ae9e74b7
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:57:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: awk "BEGIN {print \"/dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_7649722a655922fad2e75ee7af97c506 /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_7649722a655922fad2e75ee7af97c506 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:57:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_070c89e7be12165f271c2906ae9e74b7/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:57:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_b861af9cc67f5bf8d5a8f9e38f56b14e/tp_9049906389b8fcca23da885bdc1ce8ab --virtualsize 2097152K --name brick_9049906389b8fcca23da885bdc1ce8ab
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_9049906389b8fcca23da885bdc1ce8ab" created.
>[kubeexec] DEBUG 2018/06/08 08:57:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: chown :2001 /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_070c89e7be12165f271c2906ae9e74b7/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:57:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_7649722a655922fad2e75ee7af97c506 /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_7649722a655922fad2e75ee7af97c506
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:57:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_9049906389b8fcca23da885bdc1ce8ab
>Result: meta-data=/dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_9049906389b8fcca23da885bdc1ce8ab isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:57:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_7649722a655922fad2e75ee7af97c506/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:57:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: chmod 2775 /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_070c89e7be12165f271c2906ae9e74b7/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:57:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: awk "BEGIN {print \"/dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_9049906389b8fcca23da885bdc1ce8ab /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_9049906389b8fcca23da885bdc1ce8ab xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:57:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: chown :2000 /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_7649722a655922fad2e75ee7af97c506/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:57:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir -p /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_81fe2c773a661172f7014e85c2f68184
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:57:20 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: chmod 2775 /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_7649722a655922fad2e75ee7af97c506/brick
>Result: 
>[cmdexec] INFO 2018/06/08 08:57:20 Creating volume vol_ad1e5849e9566f1bcaa09cfb9c0b96ef replica 3
>[kubeexec] DEBUG 2018/06/08 08:57:20 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_9049906389b8fcca23da885bdc1ce8ab /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_9049906389b8fcca23da885bdc1ce8ab
>Result: 
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 111.168Âµs
>[kubeexec] DEBUG 2018/06/08 08:57:20 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir -p /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_d11702f97397089b0224af9ef7a122f1
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:57:20 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_9049906389b8fcca23da885bdc1ce8ab/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:57:20 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: chown :2003 /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_9049906389b8fcca23da885bdc1ce8ab/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:57:20 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_9394bc70699b006c5460c9f654cf345f/tp_81fe2c773a661172f7014e85c2f68184 --virtualsize 2097152K --name brick_81fe2c773a661172f7014e85c2f68184
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_81fe2c773a661172f7014e85c2f68184" created.
>[kubeexec] DEBUG 2018/06/08 08:57:20 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: chmod 2775 /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_9049906389b8fcca23da885bdc1ce8ab/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:57:20 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_81fe2c773a661172f7014e85c2f68184
>Result: meta-data=/dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_81fe2c773a661172f7014e85c2f68184 isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:57:20 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_3a4297677881963e3f80124971d50eea/tp_d11702f97397089b0224af9ef7a122f1 --virtualsize 2097152K --name brick_d11702f97397089b0224af9ef7a122f1
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_d11702f97397089b0224af9ef7a122f1" created.
>[kubeexec] DEBUG 2018/06/08 08:57:20 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: awk "BEGIN {print \"/dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_81fe2c773a661172f7014e85c2f68184 /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_81fe2c773a661172f7014e85c2f68184 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:57:20 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_d11702f97397089b0224af9ef7a122f1
>Result: meta-data=/dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_d11702f97397089b0224af9ef7a122f1 isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:57:20 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_81fe2c773a661172f7014e85c2f68184 /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_81fe2c773a661172f7014e85c2f68184
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:57:20 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: awk "BEGIN {print \"/dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_d11702f97397089b0224af9ef7a122f1 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_d11702f97397089b0224af9ef7a122f1 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:57:20 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_81fe2c773a661172f7014e85c2f68184/brick
>Result: 
>[negroni] Started DELETE /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Started DELETE /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 202 Accepted in 11.580995ms
>[asynchttp] INFO 2018/06/08 08:57:20 asynchttp.go:288: Started job b6d4e666dcbba47c0877c64a0f0cfb8f
>[heketi] INFO 2018/06/08 08:57:20 Started async operation: Delete Volume
>[negroni] Started GET /queue/b6d4e666dcbba47c0877c64a0f0cfb8f
>[negroni] Completed 200 OK in 117.014Âµs
>[negroni] Completed 202 Accepted in 26.777963ms
>[asynchttp] INFO 2018/06/08 08:57:20 asynchttp.go:288: Started job dc23356206ee34a3d146cb2645e3a104
>[heketi] INFO 2018/06/08 08:57:20 Started async operation: Delete Volume
>[negroni] Started GET /queue/dc23356206ee34a3d146cb2645e3a104
>[negroni] Completed 200 OK in 92.719Âµs
>[kubeexec] DEBUG 2018/06/08 08:57:20 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: chown :2003 /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_81fe2c773a661172f7014e85c2f68184/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:57:20 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_d11702f97397089b0224af9ef7a122f1 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_d11702f97397089b0224af9ef7a122f1
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:57:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: chmod 2775 /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_81fe2c773a661172f7014e85c2f68184/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:57:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_d11702f97397089b0224af9ef7a122f1/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:57:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: chown :2001 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_d11702f97397089b0224af9ef7a122f1/brick
>Result: 
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 171.094Âµs
>[kubeexec] DEBUG 2018/06/08 08:57:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: chmod 2775 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_d11702f97397089b0224af9ef7a122f1/brick
>Result: 
>[cmdexec] INFO 2018/06/08 08:57:21 Creating volume vol_15e0122e942fc41f80666a3714670682 replica 3
>[kubeexec] DEBUG 2018/06/08 08:57:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir -p /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_e4aaed99528f9d907b84f92e740f0da4
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:57:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script volume create vol_ad1e5849e9566f1bcaa09cfb9c0b96ef replica 3 10.70.47.76:/var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_94745514333fff7f22e8d60eff97d635/brick 10.70.46.122:/var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_7649722a655922fad2e75ee7af97c506/brick 10.70.46.187:/var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_a70d481dd9566d346b4d60131fb23a9c/brick
>Result: volume create: vol_ad1e5849e9566f1bcaa09cfb9c0b96ef: success: please start the volume to access data
>[kubeexec] DEBUG 2018/06/08 08:57:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_66a9af9f6bac95f9d8d556a2f14c29d3/tp_e4aaed99528f9d907b84f92e740f0da4 --virtualsize 2097152K --name brick_e4aaed99528f9d907b84f92e740f0da4
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_e4aaed99528f9d907b84f92e740f0da4" created.
>[kubeexec] DEBUG 2018/06/08 08:57:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_e4aaed99528f9d907b84f92e740f0da4
>Result: meta-data=/dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_e4aaed99528f9d907b84f92e740f0da4 isize=512    agcount=8, agsize=65536 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=524288, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 08:57:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: awk "BEGIN {print \"/dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_e4aaed99528f9d907b84f92e740f0da4 /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_e4aaed99528f9d907b84f92e740f0da4 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[negroni] Started GET /queue/b6d4e666dcbba47c0877c64a0f0cfb8f
>[negroni] Completed 200 OK in 106.764Âµs
>[negroni] Started GET /queue/dc23356206ee34a3d146cb2645e3a104
>[negroni] Completed 200 OK in 102.763Âµs
>[kubeexec] DEBUG 2018/06/08 08:57:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_e4aaed99528f9d907b84f92e740f0da4 /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_e4aaed99528f9d907b84f92e740f0da4
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:57:22 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_e4aaed99528f9d907b84f92e740f0da4/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 08:57:22 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: chown :2003 /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_e4aaed99528f9d907b84f92e740f0da4/brick
>Result: 
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 141.979Âµs
>[kubeexec] DEBUG 2018/06/08 08:57:22 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: chmod 2775 /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_e4aaed99528f9d907b84f92e740f0da4/brick
>Result: 
>[cmdexec] INFO 2018/06/08 08:57:22 Creating volume vol_b99532640a5201d243193159ee762ae4 replica 3
>[kubeexec] DEBUG 2018/06/08 08:57:22 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvremove --autobackup=n -f   vg_3a4297677881963e3f80124971d50eea/tp_00c2d7c5fda2de77f930386434235209
>
>Result:   Logical volume "tp_00c2d7c5fda2de77f930386434235209" successfully removed
>[kubeexec] DEBUG 2018/06/08 08:57:22 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script snapshot list vol_5db55c483f6984d9917cfe4b3c8b3cbc --xml
>Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
><cliOutput>
>  <opRet>-1</opRet>
>  <opErrno>30806</opErrno>
>  <opErrstr>Volume (vol_5db55c483f6984d9917cfe4b3c8b3cbc) does not exist</opErrstr>
></cliOutput>
>[negroni] Started GET /queue/b6d4e666dcbba47c0877c64a0f0cfb8f
>[negroni] Completed 200 OK in 109.234Âµs
>[negroni] Started GET /queue/dc23356206ee34a3d146cb2645e3a104
>[negroni] Completed 200 OK in 79.367Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 110.082Âµs
>[negroni] Started GET /queue/b6d4e666dcbba47c0877c64a0f0cfb8f
>[negroni] Completed 200 OK in 118.725Âµs
>[negroni] Started GET /queue/dc23356206ee34a3d146cb2645e3a104
>[negroni] Completed 200 OK in 84.982Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 115.562Âµs
>[kubeexec] DEBUG 2018/06/08 08:57:24 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script volume start vol_ad1e5849e9566f1bcaa09cfb9c0b96ef
>Result: volume start: vol_ad1e5849e9566f1bcaa09cfb9c0b96ef: success
>[kubeexec] DEBUG 2018/06/08 08:57:24 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script volume create vol_15e0122e942fc41f80666a3714670682 replica 3 10.70.46.122:/var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_d11702f97397089b0224af9ef7a122f1/brick 10.70.47.76:/var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_070c89e7be12165f271c2906ae9e74b7/brick 10.70.46.187:/var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_b6a2ea41f37c6396a2e85a6a41faa54f/brick
>Result: volume create: vol_15e0122e942fc41f80666a3714670682: success: please start the volume to access data
>[asynchttp] INFO 2018/06/08 08:57:24 asynchttp.go:292: Completed job a8d46afb5176808836420be86f6fdcd0 in 9m18.364970256s
>[kubeexec] DEBUG 2018/06/08 08:57:24 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script snapshot list vol_9c85de66c12db0a72b4d16fe888ff74d --xml
>Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
><cliOutput>
>  <opRet>-1</opRet>
>  <opErrno>30806</opErrno>
>  <opErrstr>Volume (vol_9c85de66c12db0a72b4d16fe888ff74d) does not exist</opErrstr>
></cliOutput>
>[negroni] Started GET /queue/b6d4e666dcbba47c0877c64a0f0cfb8f
>[negroni] Completed 200 OK in 130.395Âµs
>[negroni] Started GET /queue/dc23356206ee34a3d146cb2645e3a104
>[negroni] Completed 200 OK in 93.518Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 137.875Âµs
>[negroni] Started GET /queue/b6d4e666dcbba47c0877c64a0f0cfb8f
>[negroni] Completed 200 OK in 110.133Âµs
>[negroni] Started GET /queue/dc23356206ee34a3d146cb2645e3a104
>[negroni] Completed 200 OK in 135.354Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 116.697Âµs
>[negroni] Started GET /queue/b6d4e666dcbba47c0877c64a0f0cfb8f
>[negroni] Completed 200 OK in 110.827Âµs
>[negroni] Started GET /queue/dc23356206ee34a3d146cb2645e3a104
>[negroni] Completed 200 OK in 112.202Âµs
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 200 OK in 113.34Âµs
>[negroni] Started GET /queue/b6d4e666dcbba47c0877c64a0f0cfb8f
>[negroni] Completed 200 OK in 117.379Âµs
>[negroni] Started GET /queue/dc23356206ee34a3d146cb2645e3a104
>[negroni] Completed 200 OK in 109.698Âµs
>[kubeexec] DEBUG 2018/06/08 08:57:27 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script volume start vol_15e0122e942fc41f80666a3714670682
>Result: volume start: vol_15e0122e942fc41f80666a3714670682: success
>[kubeexec] DEBUG 2018/06/08 08:57:27 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script volume create vol_b99532640a5201d243193159ee762ae4 replica 3 10.70.47.76:/var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_81fe2c773a661172f7014e85c2f68184/brick 10.70.46.122:/var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_e4aaed99528f9d907b84f92e740f0da4/brick 10.70.46.187:/var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_9049906389b8fcca23da885bdc1ce8ab/brick
>Result: volume create: vol_b99532640a5201d243193159ee762ae4: success: please start the volume to access data
>[asynchttp] INFO 2018/06/08 08:57:27 asynchttp.go:292: Completed job 6158d16fba595a299f350a673e859df3 in 8m36.574982417s
>[kubeexec] DEBUG 2018/06/08 08:57:28 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: rmdir /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_00c2d7c5fda2de77f930386434235209
>Result: 
>[heketi] INFO 2018/06/08 08:57:28 Delete Volume succeeded
>[asynchttp] INFO 2018/06/08 08:57:28 asynchttp.go:292: Completed job cc7d3ffb84c36288d9c4c22cf55e3ada in 2m52.230056309s
>[negroni] Started GET /queue/cc7d3ffb84c36288d9c4c22cf55e3ada
>[negroni] Completed 204 No Content in 121.384Âµs
>[negroni] Started DELETE /volumes/966a92ed3e4374a5e634f7b133c49e52
>[negroni] Completed 404 Not Found in 2.577544ms
>[negroni] Started GET /queue/b6d4e666dcbba47c0877c64a0f0cfb8f
>[negroni] Completed 200 OK in 120.92Âµs
>[negroni] Started GET /queue/dc23356206ee34a3d146cb2645e3a104
>[negroni] Completed 200 OK in 95.77Âµs
>[negroni] Started GET /queue/b6d4e666dcbba47c0877c64a0f0cfb8f
>[negroni] Completed 200 OK in 167.322Âµs
>[negroni] Started GET /queue/dc23356206ee34a3d146cb2645e3a104
>[negroni] Completed 200 OK in 103.208Âµs
>[negroni] Started GET /queue/b6d4e666dcbba47c0877c64a0f0cfb8f
>[negroni] Completed 200 OK in 172.114Âµs
>[negroni] Started GET /queue/dc23356206ee34a3d146cb2645e3a104
>[negroni] Completed 200 OK in 123.26Âµs
>[negroni] Started GET /queue/b6d4e666dcbba47c0877c64a0f0cfb8f
>[negroni] Completed 200 OK in 133.514Âµs
>[negroni] Started GET /queue/dc23356206ee34a3d146cb2645e3a104
>[negroni] Completed 200 OK in 100.826Âµs
>[kubeexec] ERROR 2018/06/08 08:57:32 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume stop vol_5db55c483f6984d9917cfe4b3c8b3cbc force] on glusterfs-storage-pg4xc: Err[command terminated with exit code 1]: Stdout []: Stderr [volume stop: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>]
>[cmdexec] ERROR 2018/06/08 08:57:32 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:144: Unable to stop volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume stop: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[kubeexec] DEBUG 2018/06/08 08:57:32 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script volume start vol_b99532640a5201d243193159ee762ae4
>Result: volume start: vol_b99532640a5201d243193159ee762ae4: success
>[asynchttp] INFO 2018/06/08 08:57:32 asynchttp.go:292: Completed job 078bf2d1330026527a20bcfa4ebc6028 in 7m39.145694872s
>[kubeexec] ERROR 2018/06/08 08:57:32 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume stop vol_9c85de66c12db0a72b4d16fe888ff74d force] on glusterfs-storage-gxp7c: Err[command terminated with exit code 1]: Stdout []: Stderr [volume stop: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>]
>[cmdexec] ERROR 2018/06/08 08:57:32 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:144: Unable to stop volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume stop: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[kubeexec] ERROR 2018/06/08 08:57:32 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume delete vol_5db55c483f6984d9917cfe4b3c8b3cbc] on glusterfs-storage-pg4xc: Err[command terminated with exit code 1]: Stdout []: Stderr [volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>]
>[cmdexec] ERROR 2018/06/08 08:57:32 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:153: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[heketi] ERROR 2018/06/08 08:57:32 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:525: Unable to delete volume: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[heketi] ERROR 2018/06/08 08:57:32 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:429: Error executing delete volume: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[heketi] ERROR 2018/06/08 08:57:32 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:1185: Delete Volume Failed: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[asynchttp] INFO 2018/06/08 08:57:32 asynchttp.go:292: Completed job dc23356206ee34a3d146cb2645e3a104 in 11.681906846s
>[kubeexec] ERROR 2018/06/08 08:57:32 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume delete vol_9c85de66c12db0a72b4d16fe888ff74d] on glusterfs-storage-gxp7c: Err[command terminated with exit code 1]: Stdout []: Stderr [volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>]
>[cmdexec] ERROR 2018/06/08 08:57:32 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:153: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[heketi] ERROR 2018/06/08 08:57:32 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:525: Unable to delete volume: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[heketi] ERROR 2018/06/08 08:57:32 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:429: Error executing delete volume: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[heketi] ERROR 2018/06/08 08:57:32 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:1185: Delete Volume Failed: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[asynchttp] INFO 2018/06/08 08:57:32 asynchttp.go:292: Completed job b6d4e666dcbba47c0877c64a0f0cfb8f in 11.926922311s
>[negroni] Started GET /queue/b6d4e666dcbba47c0877c64a0f0cfb8f
>[negroni] Completed 500 Internal Server Error in 176.888Âµs
>[negroni] Started GET /queue/dc23356206ee34a3d146cb2645e3a104
>[negroni] Completed 500 Internal Server Error in 166.683Âµs
>[negroni] Started DELETE /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 202 Accepted in 22.79139ms
>[asynchttp] INFO 2018/06/08 08:57:50 asynchttp.go:288: Started job 2d7a1ce31edf99d77108eb3b8bcde568
>[heketi] INFO 2018/06/08 08:57:50 Started async operation: Delete Volume
>[negroni] Started GET /queue/2d7a1ce31edf99d77108eb3b8bcde568
>[negroni] Completed 200 OK in 116.265Âµs
>[kubeexec] DEBUG 2018/06/08 08:57:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script snapshot list vol_9c85de66c12db0a72b4d16fe888ff74d --xml
>Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
><cliOutput>
>  <opRet>-1</opRet>
>  <opErrno>30806</opErrno>
>  <opErrstr>Volume (vol_9c85de66c12db0a72b4d16fe888ff74d) does not exist</opErrstr>
></cliOutput>
>[kubeexec] ERROR 2018/06/08 08:57:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume stop vol_9c85de66c12db0a72b4d16fe888ff74d force] on glusterfs-storage-gxp7c: Err[command terminated with exit code 1]: Stdout []: Stderr [volume stop: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>]
>[cmdexec] ERROR 2018/06/08 08:57:51 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:144: Unable to stop volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume stop: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[kubeexec] ERROR 2018/06/08 08:57:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume delete vol_9c85de66c12db0a72b4d16fe888ff74d] on glusterfs-storage-gxp7c: Err[command terminated with exit code 1]: Stdout []: Stderr [volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>]
>[cmdexec] ERROR 2018/06/08 08:57:51 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:153: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[heketi] ERROR 2018/06/08 08:57:51 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:525: Unable to delete volume: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[heketi] ERROR 2018/06/08 08:57:51 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:429: Error executing delete volume: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[heketi] ERROR 2018/06/08 08:57:51 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:1185: Delete Volume Failed: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[asynchttp] INFO 2018/06/08 08:57:51 asynchttp.go:292: Completed job 2d7a1ce31edf99d77108eb3b8bcde568 in 757.020328ms
>[negroni] Started GET /queue/2d7a1ce31edf99d77108eb3b8bcde568
>[negroni] Completed 500 Internal Server Error in 174.482Âµs
>[negroni] Started DELETE /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 202 Accepted in 22.106422ms
>[asynchttp] INFO 2018/06/08 08:58:35 asynchttp.go:288: Started job b2e39025ac3a11ffff7a7a616f379150
>[heketi] INFO 2018/06/08 08:58:35 Started async operation: Delete Volume
>[negroni] Started GET /queue/b2e39025ac3a11ffff7a7a616f379150
>[negroni] Completed 200 OK in 149.541Âµs
>[kubeexec] DEBUG 2018/06/08 08:58:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script snapshot list vol_9c85de66c12db0a72b4d16fe888ff74d --xml
>Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
><cliOutput>
>  <opRet>-1</opRet>
>  <opErrno>30806</opErrno>
>  <opErrstr>Volume (vol_9c85de66c12db0a72b4d16fe888ff74d) does not exist</opErrstr>
></cliOutput>
>[kubeexec] ERROR 2018/06/08 08:58:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume stop vol_9c85de66c12db0a72b4d16fe888ff74d force] on glusterfs-storage-gxp7c: Err[command terminated with exit code 1]: Stdout []: Stderr [volume stop: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>]
>[cmdexec] ERROR 2018/06/08 08:58:36 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:144: Unable to stop volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume stop: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[kubeexec] ERROR 2018/06/08 08:58:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume delete vol_9c85de66c12db0a72b4d16fe888ff74d] on glusterfs-storage-gxp7c: Err[command terminated with exit code 1]: Stdout []: Stderr [volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>]
>[cmdexec] ERROR 2018/06/08 08:58:36 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:153: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[heketi] ERROR 2018/06/08 08:58:36 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:525: Unable to delete volume: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[heketi] ERROR 2018/06/08 08:58:36 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:429: Error executing delete volume: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[asynchttp] INFO 2018/06/08 08:58:36 asynchttp.go:292: Completed job b2e39025ac3a11ffff7a7a616f379150 in 813.559265ms
>[heketi] ERROR 2018/06/08 08:58:36 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:1185: Delete Volume Failed: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[negroni] Started GET /queue/b2e39025ac3a11ffff7a7a616f379150
>[negroni] Completed 500 Internal Server Error in 196.566Âµs
>[heketi] INFO 2018/06/08 08:59:14 Starting Node Health Status refresh
>[cmdexec] INFO 2018/06/08 08:59:14 Check Glusterd service status in node dhcp46-187.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 08:59:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:33 UTC; 2h 35min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48fed567_6ae4_11e8_bc4b_005056a5f18a.slice/docker-a506d9aaca4b440d184df40d01e90b2040811c8090bf4c19d892081307b28c0c.scope/system.slice/glusterd.service
>           ââ  433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1228 /usr/sbin/glusterfsd -s 10.70.46.187 --volfile-id heketidbstorage.10.70.46.187.var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.187-var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.pid -S /var/run/gluster/c08898c2e3b0190c62df3a9d4a013c92.socket --brick-name /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.log --xlator-option *-posix.glusterd-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ20535 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/8fb729a0bbd51b204491c603c84acaa4.socket --xlator-option *replicate*.node-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034
>
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Unit glusterd.service entered failed state.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: glusterd.service failed.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:33 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 08:59:14 Periodic health check status: node 278bd6b4e16a8e62ef15aaae22e6abc1 up=true
>[cmdexec] INFO 2018/06/08 08:59:14 Check Glusterd service status in node dhcp46-122.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 08:59:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:42 UTC; 2h 35min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4900c968_6ae4_11e8_bc4b_005056a5f18a.slice/docker-438fb5fb16a82a3e701b28fa1f1ef8623dc37ca06374c604e5781a2b2ab2e06d.scope/system.slice/glusterd.service
>           ââ  433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1106 /usr/sbin/glusterfsd -s 10.70.46.122 --volfile-id heketidbstorage.10.70.46.122.var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.122-var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.pid -S /var/run/gluster/9fa3ac3b12826912287adc88c26ac134.socket --brick-name /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_29e5b36d921cca1b0699871d9727c932/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.log --xlator-option *-posix.glusterd-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ21076 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f8994cdb09351d83e70da08261d99c64.socket --xlator-option *replicate*.node-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0
>
>Jun 08 06:23:40 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:42 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 08:59:14 Periodic health check status: node 70423cc0bcd044fe5ba8bbbd256a3e49 up=true
>[cmdexec] INFO 2018/06/08 08:59:14 Check Glusterd service status in node dhcp47-76.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 08:59:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 07:25:20 UTC; 1h 33min ago
>  Process: 828 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 830 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5cb2299_6aec_11e8_bc4b_005056a5f18a.slice/docker-7f9223abc9a8641cea68f374f4039235515978ee2dab970ac6cbef182c535426.scope/system.slice/glusterd.service
>           ââ  830 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1197 /usr/sbin/glusterfsd -s 10.70.47.76 --volfile-id heketidbstorage.10.70.47.76.var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.76-var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.pid -S /var/run/gluster/b6976f9ddc3abc90e1dfa91c35a7c611.socket --brick-name /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_54c06cdf52c6fe1e3a3f8a9d39cde33d/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.log --xlator-option *-posix.glusterd-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ13734 /usr/sbin/glusterfsd -s 10.70.47.76 --volfile-id vol_bf31af76ef6c54e7e8f24f4d8711cb22.10.70.47.76.var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_c6935061d1c9dbb05f316e3d87080a38-brick -p /var/run/gluster/vols/vol_bf31af76ef6c54e7e8f24f4d8711cb22/10.70.47.76-var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_c6935061d1c9dbb05f316e3d87080a38-brick.pid -S /var/run/gluster/7aabe8187d4b998d498dd5d4edbd2626.socket --brick-name /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_c6935061d1c9dbb05f316e3d87080a38/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_c6935061d1c9dbb05f316e3d87080a38-brick.log --xlator-option *-posix.glusterd-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94 --brick-port 49153 --xlator-option vol_bf31af76ef6c54e7e8f24f4d8711cb22-server.listen-port=49153
>           ââ18989 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/ec468453b11caed5259f5e1abcc65b44.socket --xlator-option *replicate*.node-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94
>
>Jun 08 07:25:17 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 07:25:20 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Jun 08 08:53:58 dhcp47-76.lab.eng.blr.redhat.com glusterd[830]: [2018-06-08 08:53:58.338441] C [rpc-clnt.c:465:rpc_clnt_fill_request_info] 0-management: cannot lookup the saved frame corresponding to xid (103)
>[heketi] INFO 2018/06/08 08:59:14 Periodic health check status: node d942fe6c0ee5691b7cc263968f97b650 up=true
>[heketi] INFO 2018/06/08 08:59:14 Cleaned 0 nodes from health cache
>[negroni] Started DELETE /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 202 Accepted in 20.121959ms
>[asynchttp] INFO 2018/06/08 08:59:35 asynchttp.go:288: Started job f5a7380365b1a7430a7906fa4bcbab8c
>[heketi] INFO 2018/06/08 08:59:35 Started async operation: Delete Volume
>[negroni] Started GET /queue/f5a7380365b1a7430a7906fa4bcbab8c
>[negroni] Completed 200 OK in 211.034Âµs
>[kubeexec] DEBUG 2018/06/08 08:59:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script snapshot list vol_5db55c483f6984d9917cfe4b3c8b3cbc --xml
>Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
><cliOutput>
>  <opRet>-1</opRet>
>  <opErrno>30806</opErrno>
>  <opErrstr>Volume (vol_5db55c483f6984d9917cfe4b3c8b3cbc) does not exist</opErrstr>
></cliOutput>
>[kubeexec] ERROR 2018/06/08 08:59:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume stop vol_5db55c483f6984d9917cfe4b3c8b3cbc force] on glusterfs-storage-pg4xc: Err[command terminated with exit code 1]: Stdout []: Stderr [volume stop: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>]
>[cmdexec] ERROR 2018/06/08 08:59:36 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:144: Unable to stop volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume stop: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[kubeexec] ERROR 2018/06/08 08:59:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume delete vol_5db55c483f6984d9917cfe4b3c8b3cbc] on glusterfs-storage-pg4xc: Err[command terminated with exit code 1]: Stdout []: Stderr [volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>]
>[cmdexec] ERROR 2018/06/08 08:59:36 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:153: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[heketi] ERROR 2018/06/08 08:59:36 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:525: Unable to delete volume: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[heketi] ERROR 2018/06/08 08:59:36 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:429: Error executing delete volume: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[heketi] ERROR 2018/06/08 08:59:36 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:1185: Delete Volume Failed: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[asynchttp] INFO 2018/06/08 08:59:36 asynchttp.go:292: Completed job f5a7380365b1a7430a7906fa4bcbab8c in 771.888072ms
>[negroni] Started GET /queue/f5a7380365b1a7430a7906fa4bcbab8c
>[negroni] Completed 500 Internal Server Error in 184.089Âµs
>[negroni] Started DELETE /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 202 Accepted in 21.372586ms
>[asynchttp] INFO 2018/06/08 08:59:50 asynchttp.go:288: Started job eb71a9488b90308975bc0f5e5c20c850
>[heketi] INFO 2018/06/08 08:59:50 Started async operation: Delete Volume
>[negroni] Started GET /queue/eb71a9488b90308975bc0f5e5c20c850
>[negroni] Completed 200 OK in 106.679Âµs
>[kubeexec] DEBUG 2018/06/08 08:59:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script snapshot list vol_9c85de66c12db0a72b4d16fe888ff74d --xml
>Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
><cliOutput>
>  <opRet>-1</opRet>
>  <opErrno>30806</opErrno>
>  <opErrstr>Volume (vol_9c85de66c12db0a72b4d16fe888ff74d) does not exist</opErrstr>
></cliOutput>
>[kubeexec] ERROR 2018/06/08 08:59:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume stop vol_9c85de66c12db0a72b4d16fe888ff74d force] on glusterfs-storage-gxp7c: Err[command terminated with exit code 1]: Stdout []: Stderr [volume stop: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>]
>[cmdexec] ERROR 2018/06/08 08:59:51 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:144: Unable to stop volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume stop: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[kubeexec] ERROR 2018/06/08 08:59:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume delete vol_9c85de66c12db0a72b4d16fe888ff74d] on glusterfs-storage-gxp7c: Err[command terminated with exit code 1]: Stdout []: Stderr [volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>]
>[cmdexec] ERROR 2018/06/08 08:59:51 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:153: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[heketi] ERROR 2018/06/08 08:59:51 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:525: Unable to delete volume: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[heketi] ERROR 2018/06/08 08:59:51 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:429: Error executing delete volume: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[heketi] ERROR 2018/06/08 08:59:51 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:1185: Delete Volume Failed: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[asynchttp] INFO 2018/06/08 08:59:51 asynchttp.go:292: Completed job eb71a9488b90308975bc0f5e5c20c850 in 747.007149ms
>[negroni] Started GET /queue/eb71a9488b90308975bc0f5e5c20c850
>[negroni] Completed 500 Internal Server Error in 175.836Âµs
>[heketi] INFO 2018/06/08 09:01:14 Starting Node Health Status refresh
>[cmdexec] INFO 2018/06/08 09:01:14 Check Glusterd service status in node dhcp46-187.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 09:01:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:33 UTC; 2h 37min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48fed567_6ae4_11e8_bc4b_005056a5f18a.slice/docker-a506d9aaca4b440d184df40d01e90b2040811c8090bf4c19d892081307b28c0c.scope/system.slice/glusterd.service
>           ââ  433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1228 /usr/sbin/glusterfsd -s 10.70.46.187 --volfile-id heketidbstorage.10.70.46.187.var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.187-var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.pid -S /var/run/gluster/c08898c2e3b0190c62df3a9d4a013c92.socket --brick-name /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.log --xlator-option *-posix.glusterd-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ20535 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/8fb729a0bbd51b204491c603c84acaa4.socket --xlator-option *replicate*.node-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034
>
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Unit glusterd.service entered failed state.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: glusterd.service failed.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:33 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 09:01:14 Periodic health check status: node 278bd6b4e16a8e62ef15aaae22e6abc1 up=true
>[cmdexec] INFO 2018/06/08 09:01:14 Check Glusterd service status in node dhcp46-122.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 09:01:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:42 UTC; 2h 37min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4900c968_6ae4_11e8_bc4b_005056a5f18a.slice/docker-438fb5fb16a82a3e701b28fa1f1ef8623dc37ca06374c604e5781a2b2ab2e06d.scope/system.slice/glusterd.service
>           ââ  433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1106 /usr/sbin/glusterfsd -s 10.70.46.122 --volfile-id heketidbstorage.10.70.46.122.var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.122-var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.pid -S /var/run/gluster/9fa3ac3b12826912287adc88c26ac134.socket --brick-name /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_29e5b36d921cca1b0699871d9727c932/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.log --xlator-option *-posix.glusterd-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ21076 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f8994cdb09351d83e70da08261d99c64.socket --xlator-option *replicate*.node-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0
>
>Jun 08 06:23:40 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:42 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 09:01:14 Periodic health check status: node 70423cc0bcd044fe5ba8bbbd256a3e49 up=true
>[cmdexec] INFO 2018/06/08 09:01:14 Check Glusterd service status in node dhcp47-76.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 09:01:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 07:25:20 UTC; 1h 35min ago
>  Process: 828 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 830 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5cb2299_6aec_11e8_bc4b_005056a5f18a.slice/docker-7f9223abc9a8641cea68f374f4039235515978ee2dab970ac6cbef182c535426.scope/system.slice/glusterd.service
>           ââ  830 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1197 /usr/sbin/glusterfsd -s 10.70.47.76 --volfile-id heketidbstorage.10.70.47.76.var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.76-var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.pid -S /var/run/gluster/b6976f9ddc3abc90e1dfa91c35a7c611.socket --brick-name /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_54c06cdf52c6fe1e3a3f8a9d39cde33d/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.log --xlator-option *-posix.glusterd-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ13734 /usr/sbin/glusterfsd -s 10.70.47.76 --volfile-id vol_bf31af76ef6c54e7e8f24f4d8711cb22.10.70.47.76.var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_c6935061d1c9dbb05f316e3d87080a38-brick -p /var/run/gluster/vols/vol_bf31af76ef6c54e7e8f24f4d8711cb22/10.70.47.76-var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_c6935061d1c9dbb05f316e3d87080a38-brick.pid -S /var/run/gluster/7aabe8187d4b998d498dd5d4edbd2626.socket --brick-name /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_c6935061d1c9dbb05f316e3d87080a38/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_c6935061d1c9dbb05f316e3d87080a38-brick.log --xlator-option *-posix.glusterd-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94 --brick-port 49153 --xlator-option vol_bf31af76ef6c54e7e8f24f4d8711cb22-server.listen-port=49153
>           ââ18989 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/ec468453b11caed5259f5e1abcc65b44.socket --xlator-option *replicate*.node-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94
>
>Jun 08 07:25:17 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 07:25:20 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Jun 08 08:53:58 dhcp47-76.lab.eng.blr.redhat.com glusterd[830]: [2018-06-08 08:53:58.338441] C [rpc-clnt.c:465:rpc_clnt_fill_request_info] 0-management: cannot lookup the saved frame corresponding to xid (103)
>[heketi] INFO 2018/06/08 09:01:14 Periodic health check status: node d942fe6c0ee5691b7cc263968f97b650 up=true
>[heketi] INFO 2018/06/08 09:01:14 Cleaned 0 nodes from health cache
>[negroni] Started DELETE /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 202 Accepted in 18.358197ms
>[asynchttp] INFO 2018/06/08 09:01:50 asynchttp.go:288: Started job e830ef7460198b6e03c5a71a4192a538
>[heketi] INFO 2018/06/08 09:01:50 Started async operation: Delete Volume
>[negroni] Started GET /queue/e830ef7460198b6e03c5a71a4192a538
>[negroni] Completed 200 OK in 145.858Âµs
>[kubeexec] DEBUG 2018/06/08 09:01:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script snapshot list vol_5db55c483f6984d9917cfe4b3c8b3cbc --xml
>Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
><cliOutput>
>  <opRet>-1</opRet>
>  <opErrno>30806</opErrno>
>  <opErrstr>Volume (vol_5db55c483f6984d9917cfe4b3c8b3cbc) does not exist</opErrstr>
></cliOutput>
>[kubeexec] ERROR 2018/06/08 09:01:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume stop vol_5db55c483f6984d9917cfe4b3c8b3cbc force] on glusterfs-storage-pg4xc: Err[command terminated with exit code 1]: Stdout []: Stderr [volume stop: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>]
>[cmdexec] ERROR 2018/06/08 09:01:51 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:144: Unable to stop volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume stop: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[kubeexec] ERROR 2018/06/08 09:01:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume delete vol_5db55c483f6984d9917cfe4b3c8b3cbc] on glusterfs-storage-pg4xc: Err[command terminated with exit code 1]: Stdout []: Stderr [volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>]
>[cmdexec] ERROR 2018/06/08 09:01:51 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:153: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[heketi] ERROR 2018/06/08 09:01:51 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:525: Unable to delete volume: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[heketi] ERROR 2018/06/08 09:01:51 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:429: Error executing delete volume: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[asynchttp] INFO 2018/06/08 09:01:51 asynchttp.go:292: Completed job e830ef7460198b6e03c5a71a4192a538 in 820.702643ms
>[heketi] ERROR 2018/06/08 09:01:51 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:1185: Delete Volume Failed: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[negroni] Started GET /queue/e830ef7460198b6e03c5a71a4192a538
>[negroni] Completed 500 Internal Server Error in 229.705Âµs
>[negroni] Started DELETE /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 202 Accepted in 20.087033ms
>[asynchttp] INFO 2018/06/08 09:02:05 asynchttp.go:288: Started job 2e462a58509319083063068ceaf27cb4
>[heketi] INFO 2018/06/08 09:02:05 Started async operation: Delete Volume
>[negroni] Started GET /queue/2e462a58509319083063068ceaf27cb4
>[negroni] Completed 200 OK in 104.781Âµs
>[kubeexec] DEBUG 2018/06/08 09:02:06 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script snapshot list vol_9c85de66c12db0a72b4d16fe888ff74d --xml
>Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
><cliOutput>
>  <opRet>-1</opRet>
>  <opErrno>30806</opErrno>
>  <opErrstr>Volume (vol_9c85de66c12db0a72b4d16fe888ff74d) does not exist</opErrstr>
></cliOutput>
>[kubeexec] ERROR 2018/06/08 09:02:06 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume stop vol_9c85de66c12db0a72b4d16fe888ff74d force] on glusterfs-storage-gxp7c: Err[command terminated with exit code 1]: Stdout []: Stderr [volume stop: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>]
>[cmdexec] ERROR 2018/06/08 09:02:06 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:144: Unable to stop volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume stop: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[kubeexec] ERROR 2018/06/08 09:02:06 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume delete vol_9c85de66c12db0a72b4d16fe888ff74d] on glusterfs-storage-gxp7c: Err[command terminated with exit code 1]: Stdout []: Stderr [volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>]
>[cmdexec] ERROR 2018/06/08 09:02:06 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:153: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[heketi] ERROR 2018/06/08 09:02:06 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:525: Unable to delete volume: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[heketi] ERROR 2018/06/08 09:02:06 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:429: Error executing delete volume: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[heketi] ERROR 2018/06/08 09:02:06 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:1185: Delete Volume Failed: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[asynchttp] INFO 2018/06/08 09:02:06 asynchttp.go:292: Completed job 2e462a58509319083063068ceaf27cb4 in 750.031888ms
>[negroni] Started GET /queue/2e462a58509319083063068ceaf27cb4
>[negroni] Completed 500 Internal Server Error in 144.254Âµs
>[heketi] INFO 2018/06/08 09:03:14 Starting Node Health Status refresh
>[cmdexec] INFO 2018/06/08 09:03:14 Check Glusterd service status in node dhcp46-187.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 09:03:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:33 UTC; 2h 39min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48fed567_6ae4_11e8_bc4b_005056a5f18a.slice/docker-a506d9aaca4b440d184df40d01e90b2040811c8090bf4c19d892081307b28c0c.scope/system.slice/glusterd.service
>           ââ  433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1228 /usr/sbin/glusterfsd -s 10.70.46.187 --volfile-id heketidbstorage.10.70.46.187.var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.187-var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.pid -S /var/run/gluster/c08898c2e3b0190c62df3a9d4a013c92.socket --brick-name /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.log --xlator-option *-posix.glusterd-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ20535 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/8fb729a0bbd51b204491c603c84acaa4.socket --xlator-option *replicate*.node-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034
>
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Unit glusterd.service entered failed state.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: glusterd.service failed.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:33 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 09:03:14 Periodic health check status: node 278bd6b4e16a8e62ef15aaae22e6abc1 up=true
>[cmdexec] INFO 2018/06/08 09:03:14 Check Glusterd service status in node dhcp46-122.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 09:03:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:42 UTC; 2h 39min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4900c968_6ae4_11e8_bc4b_005056a5f18a.slice/docker-438fb5fb16a82a3e701b28fa1f1ef8623dc37ca06374c604e5781a2b2ab2e06d.scope/system.slice/glusterd.service
>           ââ  433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1106 /usr/sbin/glusterfsd -s 10.70.46.122 --volfile-id heketidbstorage.10.70.46.122.var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.122-var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.pid -S /var/run/gluster/9fa3ac3b12826912287adc88c26ac134.socket --brick-name /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_29e5b36d921cca1b0699871d9727c932/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.log --xlator-option *-posix.glusterd-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ21076 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f8994cdb09351d83e70da08261d99c64.socket --xlator-option *replicate*.node-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0
>
>Jun 08 06:23:40 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:42 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 09:03:14 Periodic health check status: node 70423cc0bcd044fe5ba8bbbd256a3e49 up=true
>[cmdexec] INFO 2018/06/08 09:03:14 Check Glusterd service status in node dhcp47-76.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 09:03:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 07:25:20 UTC; 1h 37min ago
>  Process: 828 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 830 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5cb2299_6aec_11e8_bc4b_005056a5f18a.slice/docker-7f9223abc9a8641cea68f374f4039235515978ee2dab970ac6cbef182c535426.scope/system.slice/glusterd.service
>           ââ  830 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1197 /usr/sbin/glusterfsd -s 10.70.47.76 --volfile-id heketidbstorage.10.70.47.76.var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.76-var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.pid -S /var/run/gluster/b6976f9ddc3abc90e1dfa91c35a7c611.socket --brick-name /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_54c06cdf52c6fe1e3a3f8a9d39cde33d/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.log --xlator-option *-posix.glusterd-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ13734 /usr/sbin/glusterfsd -s 10.70.47.76 --volfile-id vol_bf31af76ef6c54e7e8f24f4d8711cb22.10.70.47.76.var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_c6935061d1c9dbb05f316e3d87080a38-brick -p /var/run/gluster/vols/vol_bf31af76ef6c54e7e8f24f4d8711cb22/10.70.47.76-var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_c6935061d1c9dbb05f316e3d87080a38-brick.pid -S /var/run/gluster/7aabe8187d4b998d498dd5d4edbd2626.socket --brick-name /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_c6935061d1c9dbb05f316e3d87080a38/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_c6935061d1c9dbb05f316e3d87080a38-brick.log --xlator-option *-posix.glusterd-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94 --brick-port 49153 --xlator-option vol_bf31af76ef6c54e7e8f24f4d8711cb22-server.listen-port=49153
>           ââ18989 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/ec468453b11caed5259f5e1abcc65b44.socket --xlator-option *replicate*.node-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94
>
>Jun 08 07:25:17 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 07:25:20 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Jun 08 08:53:58 dhcp47-76.lab.eng.blr.redhat.com glusterd[830]: [2018-06-08 08:53:58.338441] C [rpc-clnt.c:465:rpc_clnt_fill_request_info] 0-management: cannot lookup the saved frame corresponding to xid (103)
>[heketi] INFO 2018/06/08 09:03:14 Periodic health check status: node d942fe6c0ee5691b7cc263968f97b650 up=true
>[heketi] INFO 2018/06/08 09:03:14 Cleaned 0 nodes from health cache
>[negroni] Started DELETE /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 202 Accepted in 18.750892ms
>[asynchttp] INFO 2018/06/08 09:04:05 asynchttp.go:288: Started job c5ee7361d0c60e65cd4eda00b14a6cd0
>[heketi] INFO 2018/06/08 09:04:05 Started async operation: Delete Volume
>[negroni] Started GET /queue/c5ee7361d0c60e65cd4eda00b14a6cd0
>[negroni] Completed 200 OK in 138.28Âµs
>[kubeexec] DEBUG 2018/06/08 09:04:06 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script snapshot list vol_5db55c483f6984d9917cfe4b3c8b3cbc --xml
>Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
><cliOutput>
>  <opRet>-1</opRet>
>  <opErrno>30806</opErrno>
>  <opErrstr>Volume (vol_5db55c483f6984d9917cfe4b3c8b3cbc) does not exist</opErrstr>
></cliOutput>
>[kubeexec] ERROR 2018/06/08 09:04:06 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume stop vol_5db55c483f6984d9917cfe4b3c8b3cbc force] on glusterfs-storage-pg4xc: Err[command terminated with exit code 1]: Stdout []: Stderr [volume stop: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>]
>[cmdexec] ERROR 2018/06/08 09:04:06 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:144: Unable to stop volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume stop: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[kubeexec] ERROR 2018/06/08 09:04:06 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume delete vol_5db55c483f6984d9917cfe4b3c8b3cbc] on glusterfs-storage-pg4xc: Err[command terminated with exit code 1]: Stdout []: Stderr [volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>]
>[cmdexec] ERROR 2018/06/08 09:04:06 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:153: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[heketi] ERROR 2018/06/08 09:04:06 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:525: Unable to delete volume: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[heketi] ERROR 2018/06/08 09:04:06 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:429: Error executing delete volume: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[asynchttp] INFO 2018/06/08 09:04:06 asynchttp.go:292: Completed job c5ee7361d0c60e65cd4eda00b14a6cd0 in 806.347502ms
>[heketi] ERROR 2018/06/08 09:04:06 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:1185: Delete Volume Failed: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[negroni] Started GET /queue/c5ee7361d0c60e65cd4eda00b14a6cd0
>[negroni] Completed 500 Internal Server Error in 152.503Âµs
>[negroni] Started DELETE /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 202 Accepted in 26.075111ms
>[asynchttp] INFO 2018/06/08 09:04:20 asynchttp.go:288: Started job 659f5487807f00c05d2f47cb94dc9ce0
>[heketi] INFO 2018/06/08 09:04:20 Started async operation: Delete Volume
>[negroni] Started GET /queue/659f5487807f00c05d2f47cb94dc9ce0
>[negroni] Completed 200 OK in 270.265Âµs
>[kubeexec] DEBUG 2018/06/08 09:04:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script snapshot list vol_9c85de66c12db0a72b4d16fe888ff74d --xml
>Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
><cliOutput>
>  <opRet>-1</opRet>
>  <opErrno>30806</opErrno>
>  <opErrstr>Volume (vol_9c85de66c12db0a72b4d16fe888ff74d) does not exist</opErrstr>
></cliOutput>
>[kubeexec] ERROR 2018/06/08 09:04:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume stop vol_9c85de66c12db0a72b4d16fe888ff74d force] on glusterfs-storage-gxp7c: Err[command terminated with exit code 1]: Stdout []: Stderr [volume stop: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>]
>[cmdexec] ERROR 2018/06/08 09:04:21 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:144: Unable to stop volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume stop: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[kubeexec] ERROR 2018/06/08 09:04:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume delete vol_9c85de66c12db0a72b4d16fe888ff74d] on glusterfs-storage-gxp7c: Err[command terminated with exit code 1]: Stdout []: Stderr [volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>]
>[cmdexec] ERROR 2018/06/08 09:04:21 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:153: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[heketi] ERROR 2018/06/08 09:04:21 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:525: Unable to delete volume: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[heketi] ERROR 2018/06/08 09:04:21 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:429: Error executing delete volume: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[heketi] ERROR 2018/06/08 09:04:21 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:1185: Delete Volume Failed: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[asynchttp] INFO 2018/06/08 09:04:21 asynchttp.go:292: Completed job 659f5487807f00c05d2f47cb94dc9ce0 in 816.414975ms
>[negroni] Started GET /queue/659f5487807f00c05d2f47cb94dc9ce0
>[negroni] Completed 500 Internal Server Error in 242.004Âµs
>[heketi] INFO 2018/06/08 09:05:14 Starting Node Health Status refresh
>[cmdexec] INFO 2018/06/08 09:05:14 Check Glusterd service status in node dhcp46-187.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 09:05:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:33 UTC; 2h 41min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48fed567_6ae4_11e8_bc4b_005056a5f18a.slice/docker-a506d9aaca4b440d184df40d01e90b2040811c8090bf4c19d892081307b28c0c.scope/system.slice/glusterd.service
>           ââ  433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1228 /usr/sbin/glusterfsd -s 10.70.46.187 --volfile-id heketidbstorage.10.70.46.187.var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.187-var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.pid -S /var/run/gluster/c08898c2e3b0190c62df3a9d4a013c92.socket --brick-name /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.log --xlator-option *-posix.glusterd-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ20535 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/8fb729a0bbd51b204491c603c84acaa4.socket --xlator-option *replicate*.node-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034
>
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Unit glusterd.service entered failed state.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: glusterd.service failed.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:33 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 09:05:14 Periodic health check status: node 278bd6b4e16a8e62ef15aaae22e6abc1 up=true
>[cmdexec] INFO 2018/06/08 09:05:14 Check Glusterd service status in node dhcp46-122.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 09:05:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:42 UTC; 2h 41min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4900c968_6ae4_11e8_bc4b_005056a5f18a.slice/docker-438fb5fb16a82a3e701b28fa1f1ef8623dc37ca06374c604e5781a2b2ab2e06d.scope/system.slice/glusterd.service
>           ââ  433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1106 /usr/sbin/glusterfsd -s 10.70.46.122 --volfile-id heketidbstorage.10.70.46.122.var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.122-var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.pid -S /var/run/gluster/9fa3ac3b12826912287adc88c26ac134.socket --brick-name /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_29e5b36d921cca1b0699871d9727c932/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.log --xlator-option *-posix.glusterd-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ21076 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f8994cdb09351d83e70da08261d99c64.socket --xlator-option *replicate*.node-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0
>
>Jun 08 06:23:40 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:42 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 09:05:14 Periodic health check status: node 70423cc0bcd044fe5ba8bbbd256a3e49 up=true
>[cmdexec] INFO 2018/06/08 09:05:14 Check Glusterd service status in node dhcp47-76.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 09:05:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 07:25:20 UTC; 1h 39min ago
>  Process: 828 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 830 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5cb2299_6aec_11e8_bc4b_005056a5f18a.slice/docker-7f9223abc9a8641cea68f374f4039235515978ee2dab970ac6cbef182c535426.scope/system.slice/glusterd.service
>           ââ  830 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1197 /usr/sbin/glusterfsd -s 10.70.47.76 --volfile-id heketidbstorage.10.70.47.76.var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.76-var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.pid -S /var/run/gluster/b6976f9ddc3abc90e1dfa91c35a7c611.socket --brick-name /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_54c06cdf52c6fe1e3a3f8a9d39cde33d/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.log --xlator-option *-posix.glusterd-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ13734 /usr/sbin/glusterfsd -s 10.70.47.76 --volfile-id vol_bf31af76ef6c54e7e8f24f4d8711cb22.10.70.47.76.var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_c6935061d1c9dbb05f316e3d87080a38-brick -p /var/run/gluster/vols/vol_bf31af76ef6c54e7e8f24f4d8711cb22/10.70.47.76-var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_c6935061d1c9dbb05f316e3d87080a38-brick.pid -S /var/run/gluster/7aabe8187d4b998d498dd5d4edbd2626.socket --brick-name /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_c6935061d1c9dbb05f316e3d87080a38/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_c6935061d1c9dbb05f316e3d87080a38-brick.log --xlator-option *-posix.glusterd-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94 --brick-port 49153 --xlator-option vol_bf31af76ef6c54e7e8f24f4d8711cb22-server.listen-port=49153
>           ââ18989 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/ec468453b11caed5259f5e1abcc65b44.socket --xlator-option *replicate*.node-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94
>
>Jun 08 07:25:17 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 07:25:20 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Jun 08 08:53:58 dhcp47-76.lab.eng.blr.redhat.com glusterd[830]: [2018-06-08 08:53:58.338441] C [rpc-clnt.c:465:rpc_clnt_fill_request_info] 0-management: cannot lookup the saved frame corresponding to xid (103)
>[heketi] INFO 2018/06/08 09:05:14 Periodic health check status: node d942fe6c0ee5691b7cc263968f97b650 up=true
>[heketi] INFO 2018/06/08 09:05:14 Cleaned 0 nodes from health cache
>[negroni] Started DELETE /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 202 Accepted in 37.727617ms
>[asynchttp] INFO 2018/06/08 09:06:20 asynchttp.go:288: Started job 1bb8013f2d0b808f8d23139575c059bc
>[heketi] INFO 2018/06/08 09:06:20 Started async operation: Delete Volume
>[negroni] Started GET /queue/1bb8013f2d0b808f8d23139575c059bc
>[negroni] Completed 200 OK in 277.362Âµs
>[kubeexec] DEBUG 2018/06/08 09:06:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script snapshot list vol_5db55c483f6984d9917cfe4b3c8b3cbc --xml
>Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
><cliOutput>
>  <opRet>-1</opRet>
>  <opErrno>30806</opErrno>
>  <opErrstr>Volume (vol_5db55c483f6984d9917cfe4b3c8b3cbc) does not exist</opErrstr>
></cliOutput>
>[kubeexec] ERROR 2018/06/08 09:06:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume stop vol_5db55c483f6984d9917cfe4b3c8b3cbc force] on glusterfs-storage-pg4xc: Err[command terminated with exit code 1]: Stdout []: Stderr [volume stop: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>]
>[cmdexec] ERROR 2018/06/08 09:06:21 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:144: Unable to stop volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume stop: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[kubeexec] ERROR 2018/06/08 09:06:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume delete vol_5db55c483f6984d9917cfe4b3c8b3cbc] on glusterfs-storage-pg4xc: Err[command terminated with exit code 1]: Stdout []: Stderr [volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>]
>[cmdexec] ERROR 2018/06/08 09:06:21 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:153: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[heketi] ERROR 2018/06/08 09:06:21 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:525: Unable to delete volume: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[heketi] ERROR 2018/06/08 09:06:21 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:429: Error executing delete volume: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[asynchttp] INFO 2018/06/08 09:06:21 asynchttp.go:292: Completed job 1bb8013f2d0b808f8d23139575c059bc in 797.286483ms
>[heketi] ERROR 2018/06/08 09:06:21 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:1185: Delete Volume Failed: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[negroni] Started GET /queue/1bb8013f2d0b808f8d23139575c059bc
>[negroni] Completed 500 Internal Server Error in 201.559Âµs
>[negroni] Started DELETE /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 202 Accepted in 24.2101ms
>[asynchttp] INFO 2018/06/08 09:06:35 asynchttp.go:288: Started job b171bff8ed623ff2ea4a417bad8f11b4
>[heketi] INFO 2018/06/08 09:06:35 Started async operation: Delete Volume
>[negroni] Started GET /queue/b171bff8ed623ff2ea4a417bad8f11b4
>[negroni] Completed 200 OK in 137.465Âµs
>[kubeexec] DEBUG 2018/06/08 09:06:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script snapshot list vol_9c85de66c12db0a72b4d16fe888ff74d --xml
>Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
><cliOutput>
>  <opRet>-1</opRet>
>  <opErrno>30806</opErrno>
>  <opErrstr>Volume (vol_9c85de66c12db0a72b4d16fe888ff74d) does not exist</opErrstr>
></cliOutput>
>[kubeexec] ERROR 2018/06/08 09:06:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume stop vol_9c85de66c12db0a72b4d16fe888ff74d force] on glusterfs-storage-gxp7c: Err[command terminated with exit code 1]: Stdout []: Stderr [volume stop: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>]
>[cmdexec] ERROR 2018/06/08 09:06:36 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:144: Unable to stop volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume stop: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[kubeexec] ERROR 2018/06/08 09:06:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume delete vol_9c85de66c12db0a72b4d16fe888ff74d] on glusterfs-storage-gxp7c: Err[command terminated with exit code 1]: Stdout []: Stderr [volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>]
>[cmdexec] ERROR 2018/06/08 09:06:36 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:153: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[heketi] ERROR 2018/06/08 09:06:36 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:525: Unable to delete volume: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[heketi] ERROR 2018/06/08 09:06:36 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:429: Error executing delete volume: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[asynchttp] INFO 2018/06/08 09:06:36 asynchttp.go:292: Completed job b171bff8ed623ff2ea4a417bad8f11b4 in 808.326604ms
>[heketi] ERROR 2018/06/08 09:06:36 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:1185: Delete Volume Failed: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[negroni] Started GET /queue/b171bff8ed623ff2ea4a417bad8f11b4
>[negroni] Completed 500 Internal Server Error in 257.561Âµs
>[heketi] INFO 2018/06/08 09:07:14 Starting Node Health Status refresh
>[cmdexec] INFO 2018/06/08 09:07:14 Check Glusterd service status in node dhcp46-187.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 09:07:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:33 UTC; 2h 43min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48fed567_6ae4_11e8_bc4b_005056a5f18a.slice/docker-a506d9aaca4b440d184df40d01e90b2040811c8090bf4c19d892081307b28c0c.scope/system.slice/glusterd.service
>           ââ  433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1228 /usr/sbin/glusterfsd -s 10.70.46.187 --volfile-id heketidbstorage.10.70.46.187.var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.187-var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.pid -S /var/run/gluster/c08898c2e3b0190c62df3a9d4a013c92.socket --brick-name /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.log --xlator-option *-posix.glusterd-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ20535 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/8fb729a0bbd51b204491c603c84acaa4.socket --xlator-option *replicate*.node-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034
>
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Unit glusterd.service entered failed state.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: glusterd.service failed.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:33 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 09:07:14 Periodic health check status: node 278bd6b4e16a8e62ef15aaae22e6abc1 up=true
>[cmdexec] INFO 2018/06/08 09:07:14 Check Glusterd service status in node dhcp46-122.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 09:07:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:42 UTC; 2h 43min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4900c968_6ae4_11e8_bc4b_005056a5f18a.slice/docker-438fb5fb16a82a3e701b28fa1f1ef8623dc37ca06374c604e5781a2b2ab2e06d.scope/system.slice/glusterd.service
>           ââ  433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1106 /usr/sbin/glusterfsd -s 10.70.46.122 --volfile-id heketidbstorage.10.70.46.122.var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.122-var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.pid -S /var/run/gluster/9fa3ac3b12826912287adc88c26ac134.socket --brick-name /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_29e5b36d921cca1b0699871d9727c932/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.log --xlator-option *-posix.glusterd-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ21076 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f8994cdb09351d83e70da08261d99c64.socket --xlator-option *replicate*.node-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0
>
>Jun 08 06:23:40 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:42 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 09:07:14 Periodic health check status: node 70423cc0bcd044fe5ba8bbbd256a3e49 up=true
>[cmdexec] INFO 2018/06/08 09:07:14 Check Glusterd service status in node dhcp47-76.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 09:07:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 07:25:20 UTC; 1h 41min ago
>  Process: 828 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 830 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5cb2299_6aec_11e8_bc4b_005056a5f18a.slice/docker-7f9223abc9a8641cea68f374f4039235515978ee2dab970ac6cbef182c535426.scope/system.slice/glusterd.service
>           ââ  830 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1197 /usr/sbin/glusterfsd -s 10.70.47.76 --volfile-id heketidbstorage.10.70.47.76.var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.76-var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.pid -S /var/run/gluster/b6976f9ddc3abc90e1dfa91c35a7c611.socket --brick-name /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_54c06cdf52c6fe1e3a3f8a9d39cde33d/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.log --xlator-option *-posix.glusterd-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ13734 /usr/sbin/glusterfsd -s 10.70.47.76 --volfile-id vol_bf31af76ef6c54e7e8f24f4d8711cb22.10.70.47.76.var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_c6935061d1c9dbb05f316e3d87080a38-brick -p /var/run/gluster/vols/vol_bf31af76ef6c54e7e8f24f4d8711cb22/10.70.47.76-var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_c6935061d1c9dbb05f316e3d87080a38-brick.pid -S /var/run/gluster/7aabe8187d4b998d498dd5d4edbd2626.socket --brick-name /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_c6935061d1c9dbb05f316e3d87080a38/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_c6935061d1c9dbb05f316e3d87080a38-brick.log --xlator-option *-posix.glusterd-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94 --brick-port 49153 --xlator-option vol_bf31af76ef6c54e7e8f24f4d8711cb22-server.listen-port=49153
>           ââ18989 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/ec468453b11caed5259f5e1abcc65b44.socket --xlator-option *replicate*.node-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94
>
>Jun 08 07:25:17 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 07:25:20 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Jun 08 08:53:58 dhcp47-76.lab.eng.blr.redhat.com glusterd[830]: [2018-06-08 08:53:58.338441] C [rpc-clnt.c:465:rpc_clnt_fill_request_info] 0-management: cannot lookup the saved frame corresponding to xid (103)
>[heketi] INFO 2018/06/08 09:07:14 Periodic health check status: node d942fe6c0ee5691b7cc263968f97b650 up=true
>[heketi] INFO 2018/06/08 09:07:14 Cleaned 0 nodes from health cache
>[negroni] Started DELETE /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 202 Accepted in 28.154398ms
>[asynchttp] INFO 2018/06/08 09:08:35 asynchttp.go:288: Started job b3820f5403380b01d609e3b2a370a792
>[heketi] INFO 2018/06/08 09:08:35 Started async operation: Delete Volume
>[negroni] Started GET /queue/b3820f5403380b01d609e3b2a370a792
>[negroni] Completed 200 OK in 200.353Âµs
>[kubeexec] DEBUG 2018/06/08 09:08:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script snapshot list vol_5db55c483f6984d9917cfe4b3c8b3cbc --xml
>Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
><cliOutput>
>  <opRet>-1</opRet>
>  <opErrno>30806</opErrno>
>  <opErrstr>Volume (vol_5db55c483f6984d9917cfe4b3c8b3cbc) does not exist</opErrstr>
></cliOutput>
>[kubeexec] ERROR 2018/06/08 09:08:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume stop vol_5db55c483f6984d9917cfe4b3c8b3cbc force] on glusterfs-storage-pg4xc: Err[command terminated with exit code 1]: Stdout []: Stderr [volume stop: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>]
>[cmdexec] ERROR 2018/06/08 09:08:36 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:144: Unable to stop volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume stop: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[kubeexec] ERROR 2018/06/08 09:08:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume delete vol_5db55c483f6984d9917cfe4b3c8b3cbc] on glusterfs-storage-pg4xc: Err[command terminated with exit code 1]: Stdout []: Stderr [volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>]
>[cmdexec] ERROR 2018/06/08 09:08:36 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:153: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[heketi] ERROR 2018/06/08 09:08:36 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:525: Unable to delete volume: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[heketi] ERROR 2018/06/08 09:08:36 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:429: Error executing delete volume: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[asynchttp] INFO 2018/06/08 09:08:36 asynchttp.go:292: Completed job b3820f5403380b01d609e3b2a370a792 in 812.932688ms
>[heketi] ERROR 2018/06/08 09:08:36 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:1185: Delete Volume Failed: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[negroni] Started GET /queue/b3820f5403380b01d609e3b2a370a792
>[negroni] Completed 500 Internal Server Error in 187.886Âµs
>[negroni] Started DELETE /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 202 Accepted in 35.194116ms
>[asynchttp] INFO 2018/06/08 09:08:50 asynchttp.go:288: Started job a1bdf9a292b43ef09b1f60ec2ad7f6fa
>[heketi] INFO 2018/06/08 09:08:50 Started async operation: Delete Volume
>[negroni] Started GET /queue/a1bdf9a292b43ef09b1f60ec2ad7f6fa
>[negroni] Completed 200 OK in 194.726Âµs
>[kubeexec] DEBUG 2018/06/08 09:08:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script snapshot list vol_9c85de66c12db0a72b4d16fe888ff74d --xml
>Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
><cliOutput>
>  <opRet>-1</opRet>
>  <opErrno>30806</opErrno>
>  <opErrstr>Volume (vol_9c85de66c12db0a72b4d16fe888ff74d) does not exist</opErrstr>
></cliOutput>
>[kubeexec] ERROR 2018/06/08 09:08:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume stop vol_9c85de66c12db0a72b4d16fe888ff74d force] on glusterfs-storage-gxp7c: Err[command terminated with exit code 1]: Stdout []: Stderr [volume stop: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>]
>[cmdexec] ERROR 2018/06/08 09:08:51 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:144: Unable to stop volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume stop: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[kubeexec] ERROR 2018/06/08 09:08:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume delete vol_9c85de66c12db0a72b4d16fe888ff74d] on glusterfs-storage-gxp7c: Err[command terminated with exit code 1]: Stdout []: Stderr [volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>]
>[cmdexec] ERROR 2018/06/08 09:08:51 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:153: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[heketi] ERROR 2018/06/08 09:08:51 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:525: Unable to delete volume: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[heketi] ERROR 2018/06/08 09:08:51 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:429: Error executing delete volume: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[asynchttp] INFO 2018/06/08 09:08:51 asynchttp.go:292: Completed job a1bdf9a292b43ef09b1f60ec2ad7f6fa in 767.13158ms
>[heketi] ERROR 2018/06/08 09:08:51 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:1185: Delete Volume Failed: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[negroni] Started GET /queue/a1bdf9a292b43ef09b1f60ec2ad7f6fa
>[negroni] Completed 500 Internal Server Error in 133.319Âµs
>[heketi] INFO 2018/06/08 09:09:14 Starting Node Health Status refresh
>[cmdexec] INFO 2018/06/08 09:09:14 Check Glusterd service status in node dhcp46-187.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 09:09:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:33 UTC; 2h 45min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48fed567_6ae4_11e8_bc4b_005056a5f18a.slice/docker-a506d9aaca4b440d184df40d01e90b2040811c8090bf4c19d892081307b28c0c.scope/system.slice/glusterd.service
>           ââ  433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1228 /usr/sbin/glusterfsd -s 10.70.46.187 --volfile-id heketidbstorage.10.70.46.187.var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.187-var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.pid -S /var/run/gluster/c08898c2e3b0190c62df3a9d4a013c92.socket --brick-name /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.log --xlator-option *-posix.glusterd-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ20535 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/8fb729a0bbd51b204491c603c84acaa4.socket --xlator-option *replicate*.node-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034
>
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Unit glusterd.service entered failed state.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: glusterd.service failed.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:33 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 09:09:14 Periodic health check status: node 278bd6b4e16a8e62ef15aaae22e6abc1 up=true
>[cmdexec] INFO 2018/06/08 09:09:14 Check Glusterd service status in node dhcp46-122.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 09:09:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:42 UTC; 2h 45min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4900c968_6ae4_11e8_bc4b_005056a5f18a.slice/docker-438fb5fb16a82a3e701b28fa1f1ef8623dc37ca06374c604e5781a2b2ab2e06d.scope/system.slice/glusterd.service
>           ââ  433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1106 /usr/sbin/glusterfsd -s 10.70.46.122 --volfile-id heketidbstorage.10.70.46.122.var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.122-var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.pid -S /var/run/gluster/9fa3ac3b12826912287adc88c26ac134.socket --brick-name /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_29e5b36d921cca1b0699871d9727c932/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.log --xlator-option *-posix.glusterd-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ21076 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f8994cdb09351d83e70da08261d99c64.socket --xlator-option *replicate*.node-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0
>
>Jun 08 06:23:40 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:42 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 09:09:14 Periodic health check status: node 70423cc0bcd044fe5ba8bbbd256a3e49 up=true
>[cmdexec] INFO 2018/06/08 09:09:14 Check Glusterd service status in node dhcp47-76.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 09:09:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 07:25:20 UTC; 1h 43min ago
>  Process: 828 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 830 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5cb2299_6aec_11e8_bc4b_005056a5f18a.slice/docker-7f9223abc9a8641cea68f374f4039235515978ee2dab970ac6cbef182c535426.scope/system.slice/glusterd.service
>           ââ  830 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1197 /usr/sbin/glusterfsd -s 10.70.47.76 --volfile-id heketidbstorage.10.70.47.76.var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.76-var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.pid -S /var/run/gluster/b6976f9ddc3abc90e1dfa91c35a7c611.socket --brick-name /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_54c06cdf52c6fe1e3a3f8a9d39cde33d/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.log --xlator-option *-posix.glusterd-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ13734 /usr/sbin/glusterfsd -s 10.70.47.76 --volfile-id vol_bf31af76ef6c54e7e8f24f4d8711cb22.10.70.47.76.var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_c6935061d1c9dbb05f316e3d87080a38-brick -p /var/run/gluster/vols/vol_bf31af76ef6c54e7e8f24f4d8711cb22/10.70.47.76-var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_c6935061d1c9dbb05f316e3d87080a38-brick.pid -S /var/run/gluster/7aabe8187d4b998d498dd5d4edbd2626.socket --brick-name /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_c6935061d1c9dbb05f316e3d87080a38/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_c6935061d1c9dbb05f316e3d87080a38-brick.log --xlator-option *-posix.glusterd-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94 --brick-port 49153 --xlator-option vol_bf31af76ef6c54e7e8f24f4d8711cb22-server.listen-port=49153
>           ââ18989 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/ec468453b11caed5259f5e1abcc65b44.socket --xlator-option *replicate*.node-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94
>
>Jun 08 07:25:17 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 07:25:20 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Jun 08 08:53:58 dhcp47-76.lab.eng.blr.redhat.com glusterd[830]: [2018-06-08 08:53:58.338441] C [rpc-clnt.c:465:rpc_clnt_fill_request_info] 0-management: cannot lookup the saved frame corresponding to xid (103)
>[heketi] INFO 2018/06/08 09:09:14 Periodic health check status: node d942fe6c0ee5691b7cc263968f97b650 up=true
>[heketi] INFO 2018/06/08 09:09:14 Cleaned 0 nodes from health cache
>[negroni] Started DELETE /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 202 Accepted in 19.103177ms
>[asynchttp] INFO 2018/06/08 09:10:50 asynchttp.go:288: Started job bc5e310459e34fcba258116adf39ff85
>[heketi] INFO 2018/06/08 09:10:50 Started async operation: Delete Volume
>[negroni] Started GET /queue/bc5e310459e34fcba258116adf39ff85
>[negroni] Completed 200 OK in 160.312Âµs
>[kubeexec] DEBUG 2018/06/08 09:10:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script snapshot list vol_5db55c483f6984d9917cfe4b3c8b3cbc --xml
>Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
><cliOutput>
>  <opRet>-1</opRet>
>  <opErrno>30806</opErrno>
>  <opErrstr>Volume (vol_5db55c483f6984d9917cfe4b3c8b3cbc) does not exist</opErrstr>
></cliOutput>
>[kubeexec] ERROR 2018/06/08 09:10:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume stop vol_5db55c483f6984d9917cfe4b3c8b3cbc force] on glusterfs-storage-pg4xc: Err[command terminated with exit code 1]: Stdout []: Stderr [volume stop: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>]
>[cmdexec] ERROR 2018/06/08 09:10:51 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:144: Unable to stop volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume stop: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[kubeexec] ERROR 2018/06/08 09:10:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume delete vol_5db55c483f6984d9917cfe4b3c8b3cbc] on glusterfs-storage-pg4xc: Err[command terminated with exit code 1]: Stdout []: Stderr [volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>]
>[cmdexec] ERROR 2018/06/08 09:10:51 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:153: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[heketi] ERROR 2018/06/08 09:10:51 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:525: Unable to delete volume: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[heketi] ERROR 2018/06/08 09:10:51 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:429: Error executing delete volume: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[heketi] ERROR 2018/06/08 09:10:51 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:1185: Delete Volume Failed: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[asynchttp] INFO 2018/06/08 09:10:51 asynchttp.go:292: Completed job bc5e310459e34fcba258116adf39ff85 in 780.127742ms
>[negroni] Started GET /queue/bc5e310459e34fcba258116adf39ff85
>[negroni] Completed 500 Internal Server Error in 127.695Âµs
>[negroni] Started DELETE /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 202 Accepted in 29.447604ms
>[asynchttp] INFO 2018/06/08 09:11:05 asynchttp.go:288: Started job 8f9a059d334e59fddb797480ae337e71
>[heketi] INFO 2018/06/08 09:11:05 Started async operation: Delete Volume
>[negroni] Started GET /queue/8f9a059d334e59fddb797480ae337e71
>[negroni] Completed 200 OK in 181.902Âµs
>[kubeexec] DEBUG 2018/06/08 09:11:06 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script snapshot list vol_9c85de66c12db0a72b4d16fe888ff74d --xml
>Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
><cliOutput>
>  <opRet>-1</opRet>
>  <opErrno>30806</opErrno>
>  <opErrstr>Volume (vol_9c85de66c12db0a72b4d16fe888ff74d) does not exist</opErrstr>
></cliOutput>
>[kubeexec] ERROR 2018/06/08 09:11:06 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume stop vol_9c85de66c12db0a72b4d16fe888ff74d force] on glusterfs-storage-gxp7c: Err[command terminated with exit code 1]: Stdout []: Stderr [volume stop: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>]
>[cmdexec] ERROR 2018/06/08 09:11:06 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:144: Unable to stop volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume stop: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[kubeexec] ERROR 2018/06/08 09:11:06 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume delete vol_9c85de66c12db0a72b4d16fe888ff74d] on glusterfs-storage-gxp7c: Err[command terminated with exit code 1]: Stdout []: Stderr [volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>]
>[cmdexec] ERROR 2018/06/08 09:11:06 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:153: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[heketi] ERROR 2018/06/08 09:11:06 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:525: Unable to delete volume: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[heketi] ERROR 2018/06/08 09:11:06 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:429: Error executing delete volume: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[heketi] ERROR 2018/06/08 09:11:06 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:1185: Delete Volume Failed: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[asynchttp] INFO 2018/06/08 09:11:06 asynchttp.go:292: Completed job 8f9a059d334e59fddb797480ae337e71 in 794.580448ms
>[negroni] Started GET /queue/8f9a059d334e59fddb797480ae337e71
>[negroni] Completed 500 Internal Server Error in 196.176Âµs
>[heketi] INFO 2018/06/08 09:11:14 Starting Node Health Status refresh
>[cmdexec] INFO 2018/06/08 09:11:14 Check Glusterd service status in node dhcp46-187.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 09:11:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:33 UTC; 2h 47min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48fed567_6ae4_11e8_bc4b_005056a5f18a.slice/docker-a506d9aaca4b440d184df40d01e90b2040811c8090bf4c19d892081307b28c0c.scope/system.slice/glusterd.service
>           ââ  433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1228 /usr/sbin/glusterfsd -s 10.70.46.187 --volfile-id heketidbstorage.10.70.46.187.var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.187-var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.pid -S /var/run/gluster/c08898c2e3b0190c62df3a9d4a013c92.socket --brick-name /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.log --xlator-option *-posix.glusterd-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ20535 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/8fb729a0bbd51b204491c603c84acaa4.socket --xlator-option *replicate*.node-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034
>
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Unit glusterd.service entered failed state.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: glusterd.service failed.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:33 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 09:11:14 Periodic health check status: node 278bd6b4e16a8e62ef15aaae22e6abc1 up=true
>[cmdexec] INFO 2018/06/08 09:11:14 Check Glusterd service status in node dhcp46-122.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 09:11:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:42 UTC; 2h 47min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4900c968_6ae4_11e8_bc4b_005056a5f18a.slice/docker-438fb5fb16a82a3e701b28fa1f1ef8623dc37ca06374c604e5781a2b2ab2e06d.scope/system.slice/glusterd.service
>           ââ  433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1106 /usr/sbin/glusterfsd -s 10.70.46.122 --volfile-id heketidbstorage.10.70.46.122.var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.122-var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.pid -S /var/run/gluster/9fa3ac3b12826912287adc88c26ac134.socket --brick-name /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_29e5b36d921cca1b0699871d9727c932/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.log --xlator-option *-posix.glusterd-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ21076 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f8994cdb09351d83e70da08261d99c64.socket --xlator-option *replicate*.node-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0
>
>Jun 08 06:23:40 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:42 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 09:11:14 Periodic health check status: node 70423cc0bcd044fe5ba8bbbd256a3e49 up=true
>[cmdexec] INFO 2018/06/08 09:11:14 Check Glusterd service status in node dhcp47-76.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 09:11:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 07:25:20 UTC; 1h 45min ago
>  Process: 828 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 830 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5cb2299_6aec_11e8_bc4b_005056a5f18a.slice/docker-7f9223abc9a8641cea68f374f4039235515978ee2dab970ac6cbef182c535426.scope/system.slice/glusterd.service
>           ââ  830 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1197 /usr/sbin/glusterfsd -s 10.70.47.76 --volfile-id heketidbstorage.10.70.47.76.var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.76-var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.pid -S /var/run/gluster/b6976f9ddc3abc90e1dfa91c35a7c611.socket --brick-name /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_54c06cdf52c6fe1e3a3f8a9d39cde33d/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.log --xlator-option *-posix.glusterd-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ13734 /usr/sbin/glusterfsd -s 10.70.47.76 --volfile-id vol_bf31af76ef6c54e7e8f24f4d8711cb22.10.70.47.76.var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_c6935061d1c9dbb05f316e3d87080a38-brick -p /var/run/gluster/vols/vol_bf31af76ef6c54e7e8f24f4d8711cb22/10.70.47.76-var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_c6935061d1c9dbb05f316e3d87080a38-brick.pid -S /var/run/gluster/7aabe8187d4b998d498dd5d4edbd2626.socket --brick-name /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_c6935061d1c9dbb05f316e3d87080a38/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_c6935061d1c9dbb05f316e3d87080a38-brick.log --xlator-option *-posix.glusterd-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94 --brick-port 49153 --xlator-option vol_bf31af76ef6c54e7e8f24f4d8711cb22-server.listen-port=49153
>           ââ18989 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/ec468453b11caed5259f5e1abcc65b44.socket --xlator-option *replicate*.node-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94
>
>Jun 08 07:25:17 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 07:25:20 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Jun 08 08:53:58 dhcp47-76.lab.eng.blr.redhat.com glusterd[830]: [2018-06-08 08:53:58.338441] C [rpc-clnt.c:465:rpc_clnt_fill_request_info] 0-management: cannot lookup the saved frame corresponding to xid (103)
>[heketi] INFO 2018/06/08 09:11:14 Periodic health check status: node d942fe6c0ee5691b7cc263968f97b650 up=true
>[heketi] INFO 2018/06/08 09:11:14 Cleaned 0 nodes from health cache
>[negroni] Started DELETE /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 202 Accepted in 21.512522ms
>[asynchttp] INFO 2018/06/08 09:13:05 asynchttp.go:288: Started job dfa50bdad7eded8f0001e1ea503a966d
>[heketi] INFO 2018/06/08 09:13:05 Started async operation: Delete Volume
>[negroni] Started GET /queue/dfa50bdad7eded8f0001e1ea503a966d
>[negroni] Completed 200 OK in 198.658Âµs
>[kubeexec] DEBUG 2018/06/08 09:13:06 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script snapshot list vol_5db55c483f6984d9917cfe4b3c8b3cbc --xml
>Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
><cliOutput>
>  <opRet>-1</opRet>
>  <opErrno>30806</opErrno>
>  <opErrstr>Volume (vol_5db55c483f6984d9917cfe4b3c8b3cbc) does not exist</opErrstr>
></cliOutput>
>[kubeexec] ERROR 2018/06/08 09:13:06 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume stop vol_5db55c483f6984d9917cfe4b3c8b3cbc force] on glusterfs-storage-pg4xc: Err[command terminated with exit code 1]: Stdout []: Stderr [volume stop: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>]
>[cmdexec] ERROR 2018/06/08 09:13:06 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:144: Unable to stop volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume stop: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[kubeexec] ERROR 2018/06/08 09:13:06 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume delete vol_5db55c483f6984d9917cfe4b3c8b3cbc] on glusterfs-storage-pg4xc: Err[command terminated with exit code 1]: Stdout []: Stderr [volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>]
>[cmdexec] ERROR 2018/06/08 09:13:06 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:153: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[heketi] ERROR 2018/06/08 09:13:06 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:525: Unable to delete volume: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[heketi] ERROR 2018/06/08 09:13:06 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:429: Error executing delete volume: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[negroni] Started GET /queue/dfa50bdad7eded8f0001e1ea503a966d
>[negroni] Completed 200 OK in 160.212Âµs
>[asynchttp] INFO 2018/06/08 09:13:06 asynchttp.go:292: Completed job dfa50bdad7eded8f0001e1ea503a966d in 1.024468828s
>[heketi] ERROR 2018/06/08 09:13:06 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:1185: Delete Volume Failed: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[negroni] Started GET /queue/dfa50bdad7eded8f0001e1ea503a966d
>[negroni] Completed 500 Internal Server Error in 143.517Âµs
>[heketi] INFO 2018/06/08 09:13:14 Starting Node Health Status refresh
>[cmdexec] INFO 2018/06/08 09:13:14 Check Glusterd service status in node dhcp46-187.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 09:13:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:33 UTC; 2h 49min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48fed567_6ae4_11e8_bc4b_005056a5f18a.slice/docker-a506d9aaca4b440d184df40d01e90b2040811c8090bf4c19d892081307b28c0c.scope/system.slice/glusterd.service
>           ââ  433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1228 /usr/sbin/glusterfsd -s 10.70.46.187 --volfile-id heketidbstorage.10.70.46.187.var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.187-var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.pid -S /var/run/gluster/c08898c2e3b0190c62df3a9d4a013c92.socket --brick-name /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.log --xlator-option *-posix.glusterd-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ20535 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/8fb729a0bbd51b204491c603c84acaa4.socket --xlator-option *replicate*.node-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034
>
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Unit glusterd.service entered failed state.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: glusterd.service failed.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:33 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 09:13:14 Periodic health check status: node 278bd6b4e16a8e62ef15aaae22e6abc1 up=true
>[cmdexec] INFO 2018/06/08 09:13:14 Check Glusterd service status in node dhcp46-122.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 09:13:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:42 UTC; 2h 49min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4900c968_6ae4_11e8_bc4b_005056a5f18a.slice/docker-438fb5fb16a82a3e701b28fa1f1ef8623dc37ca06374c604e5781a2b2ab2e06d.scope/system.slice/glusterd.service
>           ââ  433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1106 /usr/sbin/glusterfsd -s 10.70.46.122 --volfile-id heketidbstorage.10.70.46.122.var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.122-var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.pid -S /var/run/gluster/9fa3ac3b12826912287adc88c26ac134.socket --brick-name /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_29e5b36d921cca1b0699871d9727c932/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.log --xlator-option *-posix.glusterd-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ21076 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f8994cdb09351d83e70da08261d99c64.socket --xlator-option *replicate*.node-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0
>
>Jun 08 06:23:40 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:42 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 09:13:14 Periodic health check status: node 70423cc0bcd044fe5ba8bbbd256a3e49 up=true
>[cmdexec] INFO 2018/06/08 09:13:14 Check Glusterd service status in node dhcp47-76.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 09:13:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 07:25:20 UTC; 1h 47min ago
>  Process: 828 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 830 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5cb2299_6aec_11e8_bc4b_005056a5f18a.slice/docker-7f9223abc9a8641cea68f374f4039235515978ee2dab970ac6cbef182c535426.scope/system.slice/glusterd.service
>           ââ  830 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1197 /usr/sbin/glusterfsd -s 10.70.47.76 --volfile-id heketidbstorage.10.70.47.76.var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.76-var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.pid -S /var/run/gluster/b6976f9ddc3abc90e1dfa91c35a7c611.socket --brick-name /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_54c06cdf52c6fe1e3a3f8a9d39cde33d/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.log --xlator-option *-posix.glusterd-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ13734 /usr/sbin/glusterfsd -s 10.70.47.76 --volfile-id vol_bf31af76ef6c54e7e8f24f4d8711cb22.10.70.47.76.var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_c6935061d1c9dbb05f316e3d87080a38-brick -p /var/run/gluster/vols/vol_bf31af76ef6c54e7e8f24f4d8711cb22/10.70.47.76-var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_c6935061d1c9dbb05f316e3d87080a38-brick.pid -S /var/run/gluster/7aabe8187d4b998d498dd5d4edbd2626.socket --brick-name /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_c6935061d1c9dbb05f316e3d87080a38/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_c6935061d1c9dbb05f316e3d87080a38-brick.log --xlator-option *-posix.glusterd-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94 --brick-port 49153 --xlator-option vol_bf31af76ef6c54e7e8f24f4d8711cb22-server.listen-port=49153
>           ââ18989 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/ec468453b11caed5259f5e1abcc65b44.socket --xlator-option *replicate*.node-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94
>
>Jun 08 07:25:17 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 07:25:20 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Jun 08 08:53:58 dhcp47-76.lab.eng.blr.redhat.com glusterd[830]: [2018-06-08 08:53:58.338441] C [rpc-clnt.c:465:rpc_clnt_fill_request_info] 0-management: cannot lookup the saved frame corresponding to xid (103)
>[heketi] INFO 2018/06/08 09:13:14 Periodic health check status: node d942fe6c0ee5691b7cc263968f97b650 up=true
>[heketi] INFO 2018/06/08 09:13:14 Cleaned 0 nodes from health cache
>[negroni] Started DELETE /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 202 Accepted in 22.094084ms
>[asynchttp] INFO 2018/06/08 09:13:20 asynchttp.go:288: Started job e89bb0760ac4b6a89bb60546cdb7d609
>[heketi] INFO 2018/06/08 09:13:20 Started async operation: Delete Volume
>[negroni] Started GET /queue/e89bb0760ac4b6a89bb60546cdb7d609
>[negroni] Completed 200 OK in 125.19Âµs
>[kubeexec] DEBUG 2018/06/08 09:13:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script snapshot list vol_9c85de66c12db0a72b4d16fe888ff74d --xml
>Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
><cliOutput>
>  <opRet>-1</opRet>
>  <opErrno>30806</opErrno>
>  <opErrstr>Volume (vol_9c85de66c12db0a72b4d16fe888ff74d) does not exist</opErrstr>
></cliOutput>
>[kubeexec] ERROR 2018/06/08 09:13:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume stop vol_9c85de66c12db0a72b4d16fe888ff74d force] on glusterfs-storage-gxp7c: Err[command terminated with exit code 1]: Stdout []: Stderr [volume stop: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>]
>[cmdexec] ERROR 2018/06/08 09:13:21 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:144: Unable to stop volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume stop: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[kubeexec] ERROR 2018/06/08 09:13:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume delete vol_9c85de66c12db0a72b4d16fe888ff74d] on glusterfs-storage-gxp7c: Err[command terminated with exit code 1]: Stdout []: Stderr [volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>]
>[cmdexec] ERROR 2018/06/08 09:13:21 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:153: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[heketi] ERROR 2018/06/08 09:13:21 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:525: Unable to delete volume: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[heketi] ERROR 2018/06/08 09:13:21 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:429: Error executing delete volume: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[asynchttp] INFO 2018/06/08 09:13:21 asynchttp.go:292: Completed job e89bb0760ac4b6a89bb60546cdb7d609 in 748.931699ms
>[heketi] ERROR 2018/06/08 09:13:21 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:1185: Delete Volume Failed: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[negroni] Started GET /queue/e89bb0760ac4b6a89bb60546cdb7d609
>[negroni] Completed 500 Internal Server Error in 207.555Âµs
>[heketi] INFO 2018/06/08 09:15:14 Starting Node Health Status refresh
>[cmdexec] INFO 2018/06/08 09:15:14 Check Glusterd service status in node dhcp46-187.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 09:15:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:33 UTC; 2h 51min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48fed567_6ae4_11e8_bc4b_005056a5f18a.slice/docker-a506d9aaca4b440d184df40d01e90b2040811c8090bf4c19d892081307b28c0c.scope/system.slice/glusterd.service
>           ââ  433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1228 /usr/sbin/glusterfsd -s 10.70.46.187 --volfile-id heketidbstorage.10.70.46.187.var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.187-var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.pid -S /var/run/gluster/c08898c2e3b0190c62df3a9d4a013c92.socket --brick-name /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.log --xlator-option *-posix.glusterd-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ20535 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/8fb729a0bbd51b204491c603c84acaa4.socket --xlator-option *replicate*.node-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034
>
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Unit glusterd.service entered failed state.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: glusterd.service failed.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:33 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 09:15:14 Periodic health check status: node 278bd6b4e16a8e62ef15aaae22e6abc1 up=true
>[cmdexec] INFO 2018/06/08 09:15:14 Check Glusterd service status in node dhcp46-122.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 09:15:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:42 UTC; 2h 51min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4900c968_6ae4_11e8_bc4b_005056a5f18a.slice/docker-438fb5fb16a82a3e701b28fa1f1ef8623dc37ca06374c604e5781a2b2ab2e06d.scope/system.slice/glusterd.service
>           ââ  433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1106 /usr/sbin/glusterfsd -s 10.70.46.122 --volfile-id heketidbstorage.10.70.46.122.var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.122-var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.pid -S /var/run/gluster/9fa3ac3b12826912287adc88c26ac134.socket --brick-name /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_29e5b36d921cca1b0699871d9727c932/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.log --xlator-option *-posix.glusterd-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ21076 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f8994cdb09351d83e70da08261d99c64.socket --xlator-option *replicate*.node-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0
>
>Jun 08 06:23:40 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:42 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 09:15:14 Periodic health check status: node 70423cc0bcd044fe5ba8bbbd256a3e49 up=true
>[cmdexec] INFO 2018/06/08 09:15:14 Check Glusterd service status in node dhcp47-76.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 09:15:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 07:25:20 UTC; 1h 49min ago
>  Process: 828 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 830 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5cb2299_6aec_11e8_bc4b_005056a5f18a.slice/docker-7f9223abc9a8641cea68f374f4039235515978ee2dab970ac6cbef182c535426.scope/system.slice/glusterd.service
>           ââ  830 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1197 /usr/sbin/glusterfsd -s 10.70.47.76 --volfile-id heketidbstorage.10.70.47.76.var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.76-var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.pid -S /var/run/gluster/b6976f9ddc3abc90e1dfa91c35a7c611.socket --brick-name /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_54c06cdf52c6fe1e3a3f8a9d39cde33d/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.log --xlator-option *-posix.glusterd-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ13734 /usr/sbin/glusterfsd -s 10.70.47.76 --volfile-id vol_bf31af76ef6c54e7e8f24f4d8711cb22.10.70.47.76.var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_c6935061d1c9dbb05f316e3d87080a38-brick -p /var/run/gluster/vols/vol_bf31af76ef6c54e7e8f24f4d8711cb22/10.70.47.76-var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_c6935061d1c9dbb05f316e3d87080a38-brick.pid -S /var/run/gluster/7aabe8187d4b998d498dd5d4edbd2626.socket --brick-name /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_c6935061d1c9dbb05f316e3d87080a38/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_c6935061d1c9dbb05f316e3d87080a38-brick.log --xlator-option *-posix.glusterd-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94 --brick-port 49153 --xlator-option vol_bf31af76ef6c54e7e8f24f4d8711cb22-server.listen-port=49153
>           ââ18989 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/ec468453b11caed5259f5e1abcc65b44.socket --xlator-option *replicate*.node-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94
>
>Jun 08 07:25:17 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 07:25:20 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Jun 08 08:53:58 dhcp47-76.lab.eng.blr.redhat.com glusterd[830]: [2018-06-08 08:53:58.338441] C [rpc-clnt.c:465:rpc_clnt_fill_request_info] 0-management: cannot lookup the saved frame corresponding to xid (103)
>[heketi] INFO 2018/06/08 09:15:14 Periodic health check status: node d942fe6c0ee5691b7cc263968f97b650 up=true
>[heketi] INFO 2018/06/08 09:15:14 Cleaned 0 nodes from health cache
>[negroni] Started DELETE /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 202 Accepted in 29.246643ms
>[asynchttp] INFO 2018/06/08 09:15:20 asynchttp.go:288: Started job 760396b8378841f7322d46bc920f5d4c
>[heketi] INFO 2018/06/08 09:15:20 Started async operation: Delete Volume
>[negroni] Started GET /queue/760396b8378841f7322d46bc920f5d4c
>[negroni] Completed 200 OK in 216.878Âµs
>[kubeexec] DEBUG 2018/06/08 09:15:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script snapshot list vol_5db55c483f6984d9917cfe4b3c8b3cbc --xml
>Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
><cliOutput>
>  <opRet>-1</opRet>
>  <opErrno>30806</opErrno>
>  <opErrstr>Volume (vol_5db55c483f6984d9917cfe4b3c8b3cbc) does not exist</opErrstr>
></cliOutput>
>[kubeexec] ERROR 2018/06/08 09:15:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume stop vol_5db55c483f6984d9917cfe4b3c8b3cbc force] on glusterfs-storage-pg4xc: Err[command terminated with exit code 1]: Stdout []: Stderr [volume stop: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>]
>[cmdexec] ERROR 2018/06/08 09:15:21 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:144: Unable to stop volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume stop: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[kubeexec] ERROR 2018/06/08 09:15:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume delete vol_5db55c483f6984d9917cfe4b3c8b3cbc] on glusterfs-storage-pg4xc: Err[command terminated with exit code 1]: Stdout []: Stderr [volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>]
>[cmdexec] ERROR 2018/06/08 09:15:21 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:153: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[heketi] ERROR 2018/06/08 09:15:21 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:525: Unable to delete volume: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[heketi] ERROR 2018/06/08 09:15:21 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:429: Error executing delete volume: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[asynchttp] INFO 2018/06/08 09:15:21 asynchttp.go:292: Completed job 760396b8378841f7322d46bc920f5d4c in 778.675918ms
>[heketi] ERROR 2018/06/08 09:15:21 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:1185: Delete Volume Failed: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[negroni] Started GET /queue/760396b8378841f7322d46bc920f5d4c
>[negroni] Completed 500 Internal Server Error in 199.251Âµs
>[negroni] Started DELETE /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 202 Accepted in 42.510563ms
>[asynchttp] INFO 2018/06/08 09:15:35 asynchttp.go:288: Started job b3d15906db427e0a5b0864aa90ed15a1
>[heketi] INFO 2018/06/08 09:15:35 Started async operation: Delete Volume
>[negroni] Started GET /queue/b3d15906db427e0a5b0864aa90ed15a1
>[negroni] Completed 200 OK in 193.702Âµs
>[kubeexec] DEBUG 2018/06/08 09:15:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script snapshot list vol_9c85de66c12db0a72b4d16fe888ff74d --xml
>Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
><cliOutput>
>  <opRet>-1</opRet>
>  <opErrno>30806</opErrno>
>  <opErrstr>Volume (vol_9c85de66c12db0a72b4d16fe888ff74d) does not exist</opErrstr>
></cliOutput>
>[kubeexec] ERROR 2018/06/08 09:15:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume stop vol_9c85de66c12db0a72b4d16fe888ff74d force] on glusterfs-storage-gxp7c: Err[command terminated with exit code 1]: Stdout []: Stderr [volume stop: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>]
>[cmdexec] ERROR 2018/06/08 09:15:36 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:144: Unable to stop volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume stop: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[kubeexec] ERROR 2018/06/08 09:15:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume delete vol_9c85de66c12db0a72b4d16fe888ff74d] on glusterfs-storage-gxp7c: Err[command terminated with exit code 1]: Stdout []: Stderr [volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>]
>[cmdexec] ERROR 2018/06/08 09:15:36 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:153: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[heketi] ERROR 2018/06/08 09:15:36 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:525: Unable to delete volume: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[heketi] ERROR 2018/06/08 09:15:36 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:429: Error executing delete volume: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[heketi] ERROR 2018/06/08 09:15:36 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:1185: Delete Volume Failed: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[asynchttp] INFO 2018/06/08 09:15:36 asynchttp.go:292: Completed job b3d15906db427e0a5b0864aa90ed15a1 in 745.212334ms
>[negroni] Started GET /queue/b3d15906db427e0a5b0864aa90ed15a1
>[negroni] Completed 500 Internal Server Error in 318.424Âµs
>[heketi] INFO 2018/06/08 09:17:14 Starting Node Health Status refresh
>[cmdexec] INFO 2018/06/08 09:17:14 Check Glusterd service status in node dhcp46-187.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 09:17:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:33 UTC; 2h 53min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48fed567_6ae4_11e8_bc4b_005056a5f18a.slice/docker-a506d9aaca4b440d184df40d01e90b2040811c8090bf4c19d892081307b28c0c.scope/system.slice/glusterd.service
>           ââ  433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1228 /usr/sbin/glusterfsd -s 10.70.46.187 --volfile-id heketidbstorage.10.70.46.187.var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.187-var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.pid -S /var/run/gluster/c08898c2e3b0190c62df3a9d4a013c92.socket --brick-name /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.log --xlator-option *-posix.glusterd-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ20535 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/8fb729a0bbd51b204491c603c84acaa4.socket --xlator-option *replicate*.node-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034
>
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Unit glusterd.service entered failed state.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: glusterd.service failed.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:33 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 09:17:14 Periodic health check status: node 278bd6b4e16a8e62ef15aaae22e6abc1 up=true
>[cmdexec] INFO 2018/06/08 09:17:14 Check Glusterd service status in node dhcp46-122.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 09:17:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:42 UTC; 2h 53min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4900c968_6ae4_11e8_bc4b_005056a5f18a.slice/docker-438fb5fb16a82a3e701b28fa1f1ef8623dc37ca06374c604e5781a2b2ab2e06d.scope/system.slice/glusterd.service
>           ââ  433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1106 /usr/sbin/glusterfsd -s 10.70.46.122 --volfile-id heketidbstorage.10.70.46.122.var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.122-var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.pid -S /var/run/gluster/9fa3ac3b12826912287adc88c26ac134.socket --brick-name /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_29e5b36d921cca1b0699871d9727c932/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.log --xlator-option *-posix.glusterd-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ21076 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f8994cdb09351d83e70da08261d99c64.socket --xlator-option *replicate*.node-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0
>
>Jun 08 06:23:40 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:42 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 09:17:14 Periodic health check status: node 70423cc0bcd044fe5ba8bbbd256a3e49 up=true
>[cmdexec] INFO 2018/06/08 09:17:14 Check Glusterd service status in node dhcp47-76.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 09:17:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 07:25:20 UTC; 1h 51min ago
>  Process: 828 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 830 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5cb2299_6aec_11e8_bc4b_005056a5f18a.slice/docker-7f9223abc9a8641cea68f374f4039235515978ee2dab970ac6cbef182c535426.scope/system.slice/glusterd.service
>           ââ  830 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1197 /usr/sbin/glusterfsd -s 10.70.47.76 --volfile-id heketidbstorage.10.70.47.76.var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.76-var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.pid -S /var/run/gluster/b6976f9ddc3abc90e1dfa91c35a7c611.socket --brick-name /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_54c06cdf52c6fe1e3a3f8a9d39cde33d/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.log --xlator-option *-posix.glusterd-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ13734 /usr/sbin/glusterfsd -s 10.70.47.76 --volfile-id vol_bf31af76ef6c54e7e8f24f4d8711cb22.10.70.47.76.var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_c6935061d1c9dbb05f316e3d87080a38-brick -p /var/run/gluster/vols/vol_bf31af76ef6c54e7e8f24f4d8711cb22/10.70.47.76-var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_c6935061d1c9dbb05f316e3d87080a38-brick.pid -S /var/run/gluster/7aabe8187d4b998d498dd5d4edbd2626.socket --brick-name /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_c6935061d1c9dbb05f316e3d87080a38/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_c6935061d1c9dbb05f316e3d87080a38-brick.log --xlator-option *-posix.glusterd-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94 --brick-port 49153 --xlator-option vol_bf31af76ef6c54e7e8f24f4d8711cb22-server.listen-port=49153
>           ââ18989 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/ec468453b11caed5259f5e1abcc65b44.socket --xlator-option *replicate*.node-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94
>
>Jun 08 07:25:17 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 07:25:20 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Jun 08 08:53:58 dhcp47-76.lab.eng.blr.redhat.com glusterd[830]: [2018-06-08 08:53:58.338441] C [rpc-clnt.c:465:rpc_clnt_fill_request_info] 0-management: cannot lookup the saved frame corresponding to xid (103)
>[heketi] INFO 2018/06/08 09:17:14 Periodic health check status: node d942fe6c0ee5691b7cc263968f97b650 up=true
>[heketi] INFO 2018/06/08 09:17:14 Cleaned 0 nodes from health cache
>[negroni] Started DELETE /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 202 Accepted in 25.289857ms
>[asynchttp] INFO 2018/06/08 09:17:35 asynchttp.go:288: Started job 5eaabce77f4632aab4be6741b7f05da1
>[heketi] INFO 2018/06/08 09:17:35 Started async operation: Delete Volume
>[negroni] Started GET /queue/5eaabce77f4632aab4be6741b7f05da1
>[negroni] Completed 200 OK in 116.773Âµs
>[kubeexec] DEBUG 2018/06/08 09:17:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script snapshot list vol_5db55c483f6984d9917cfe4b3c8b3cbc --xml
>Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
><cliOutput>
>  <opRet>-1</opRet>
>  <opErrno>30806</opErrno>
>  <opErrstr>Volume (vol_5db55c483f6984d9917cfe4b3c8b3cbc) does not exist</opErrstr>
></cliOutput>
>[kubeexec] ERROR 2018/06/08 09:17:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume stop vol_5db55c483f6984d9917cfe4b3c8b3cbc force] on glusterfs-storage-pg4xc: Err[command terminated with exit code 1]: Stdout []: Stderr [volume stop: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>]
>[cmdexec] ERROR 2018/06/08 09:17:36 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:144: Unable to stop volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume stop: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[kubeexec] ERROR 2018/06/08 09:17:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume delete vol_5db55c483f6984d9917cfe4b3c8b3cbc] on glusterfs-storage-pg4xc: Err[command terminated with exit code 1]: Stdout []: Stderr [volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>]
>[cmdexec] ERROR 2018/06/08 09:17:36 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:153: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[heketi] ERROR 2018/06/08 09:17:36 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:525: Unable to delete volume: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[heketi] ERROR 2018/06/08 09:17:36 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:429: Error executing delete volume: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[heketi] ERROR 2018/06/08 09:17:36 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:1185: Delete Volume Failed: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[asynchttp] INFO 2018/06/08 09:17:36 asynchttp.go:292: Completed job 5eaabce77f4632aab4be6741b7f05da1 in 752.531389ms
>[negroni] Started GET /queue/5eaabce77f4632aab4be6741b7f05da1
>[negroni] Completed 500 Internal Server Error in 241.619Âµs
>[negroni] Started DELETE /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 202 Accepted in 22.642307ms
>[asynchttp] INFO 2018/06/08 09:17:50 asynchttp.go:288: Started job accc37f885102c8e16bf64fc130baa05
>[heketi] INFO 2018/06/08 09:17:50 Started async operation: Delete Volume
>[negroni] Started GET /queue/accc37f885102c8e16bf64fc130baa05
>[negroni] Completed 200 OK in 159.331Âµs
>[kubeexec] DEBUG 2018/06/08 09:17:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script snapshot list vol_9c85de66c12db0a72b4d16fe888ff74d --xml
>Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
><cliOutput>
>  <opRet>-1</opRet>
>  <opErrno>30806</opErrno>
>  <opErrstr>Volume (vol_9c85de66c12db0a72b4d16fe888ff74d) does not exist</opErrstr>
></cliOutput>
>[kubeexec] ERROR 2018/06/08 09:17:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume stop vol_9c85de66c12db0a72b4d16fe888ff74d force] on glusterfs-storage-gxp7c: Err[command terminated with exit code 1]: Stdout []: Stderr [volume stop: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>]
>[cmdexec] ERROR 2018/06/08 09:17:51 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:144: Unable to stop volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume stop: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[kubeexec] ERROR 2018/06/08 09:17:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume delete vol_9c85de66c12db0a72b4d16fe888ff74d] on glusterfs-storage-gxp7c: Err[command terminated with exit code 1]: Stdout []: Stderr [volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>]
>[cmdexec] ERROR 2018/06/08 09:17:51 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:153: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[heketi] ERROR 2018/06/08 09:17:51 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:525: Unable to delete volume: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[heketi] ERROR 2018/06/08 09:17:51 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:429: Error executing delete volume: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[asynchttp] INFO 2018/06/08 09:17:51 asynchttp.go:292: Completed job accc37f885102c8e16bf64fc130baa05 in 801.482539ms
>[heketi] ERROR 2018/06/08 09:17:51 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:1185: Delete Volume Failed: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[negroni] Started GET /queue/accc37f885102c8e16bf64fc130baa05
>[negroni] Completed 500 Internal Server Error in 129.086Âµs
>[heketi] INFO 2018/06/08 09:19:14 Starting Node Health Status refresh
>[cmdexec] INFO 2018/06/08 09:19:14 Check Glusterd service status in node dhcp46-187.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 09:19:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:33 UTC; 2h 55min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48fed567_6ae4_11e8_bc4b_005056a5f18a.slice/docker-a506d9aaca4b440d184df40d01e90b2040811c8090bf4c19d892081307b28c0c.scope/system.slice/glusterd.service
>           ââ  433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1228 /usr/sbin/glusterfsd -s 10.70.46.187 --volfile-id heketidbstorage.10.70.46.187.var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.187-var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.pid -S /var/run/gluster/c08898c2e3b0190c62df3a9d4a013c92.socket --brick-name /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.log --xlator-option *-posix.glusterd-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ20535 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/8fb729a0bbd51b204491c603c84acaa4.socket --xlator-option *replicate*.node-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034
>
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Unit glusterd.service entered failed state.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: glusterd.service failed.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:33 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 09:19:14 Periodic health check status: node 278bd6b4e16a8e62ef15aaae22e6abc1 up=true
>[cmdexec] INFO 2018/06/08 09:19:14 Check Glusterd service status in node dhcp46-122.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 09:19:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:42 UTC; 2h 55min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4900c968_6ae4_11e8_bc4b_005056a5f18a.slice/docker-438fb5fb16a82a3e701b28fa1f1ef8623dc37ca06374c604e5781a2b2ab2e06d.scope/system.slice/glusterd.service
>           ââ  433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1106 /usr/sbin/glusterfsd -s 10.70.46.122 --volfile-id heketidbstorage.10.70.46.122.var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.122-var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.pid -S /var/run/gluster/9fa3ac3b12826912287adc88c26ac134.socket --brick-name /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_29e5b36d921cca1b0699871d9727c932/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.log --xlator-option *-posix.glusterd-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ21076 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f8994cdb09351d83e70da08261d99c64.socket --xlator-option *replicate*.node-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0
>
>Jun 08 06:23:40 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:42 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 09:19:14 Periodic health check status: node 70423cc0bcd044fe5ba8bbbd256a3e49 up=true
>[cmdexec] INFO 2018/06/08 09:19:14 Check Glusterd service status in node dhcp47-76.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 09:19:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 07:25:20 UTC; 1h 53min ago
>  Process: 828 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 830 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5cb2299_6aec_11e8_bc4b_005056a5f18a.slice/docker-7f9223abc9a8641cea68f374f4039235515978ee2dab970ac6cbef182c535426.scope/system.slice/glusterd.service
>           ââ  830 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1197 /usr/sbin/glusterfsd -s 10.70.47.76 --volfile-id heketidbstorage.10.70.47.76.var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.76-var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.pid -S /var/run/gluster/b6976f9ddc3abc90e1dfa91c35a7c611.socket --brick-name /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_54c06cdf52c6fe1e3a3f8a9d39cde33d/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.log --xlator-option *-posix.glusterd-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ13734 /usr/sbin/glusterfsd -s 10.70.47.76 --volfile-id vol_bf31af76ef6c54e7e8f24f4d8711cb22.10.70.47.76.var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_c6935061d1c9dbb05f316e3d87080a38-brick -p /var/run/gluster/vols/vol_bf31af76ef6c54e7e8f24f4d8711cb22/10.70.47.76-var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_c6935061d1c9dbb05f316e3d87080a38-brick.pid -S /var/run/gluster/7aabe8187d4b998d498dd5d4edbd2626.socket --brick-name /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_c6935061d1c9dbb05f316e3d87080a38/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_c6935061d1c9dbb05f316e3d87080a38-brick.log --xlator-option *-posix.glusterd-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94 --brick-port 49153 --xlator-option vol_bf31af76ef6c54e7e8f24f4d8711cb22-server.listen-port=49153
>           ââ18989 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/ec468453b11caed5259f5e1abcc65b44.socket --xlator-option *replicate*.node-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94
>
>Jun 08 07:25:17 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 07:25:20 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Jun 08 08:53:58 dhcp47-76.lab.eng.blr.redhat.com glusterd[830]: [2018-06-08 08:53:58.338441] C [rpc-clnt.c:465:rpc_clnt_fill_request_info] 0-management: cannot lookup the saved frame corresponding to xid (103)
>[heketi] INFO 2018/06/08 09:19:14 Periodic health check status: node d942fe6c0ee5691b7cc263968f97b650 up=true
>[heketi] INFO 2018/06/08 09:19:14 Cleaned 0 nodes from health cache
>[negroni] Started DELETE /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 202 Accepted in 37.664927ms
>[asynchttp] INFO 2018/06/08 09:19:50 asynchttp.go:288: Started job a87500ed17aaa8058a96f89376065a33
>[heketi] INFO 2018/06/08 09:19:50 Started async operation: Delete Volume
>[negroni] Started GET /queue/a87500ed17aaa8058a96f89376065a33
>[negroni] Completed 200 OK in 140.491Âµs
>[kubeexec] DEBUG 2018/06/08 09:19:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script snapshot list vol_5db55c483f6984d9917cfe4b3c8b3cbc --xml
>Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
><cliOutput>
>  <opRet>-1</opRet>
>  <opErrno>30806</opErrno>
>  <opErrstr>Volume (vol_5db55c483f6984d9917cfe4b3c8b3cbc) does not exist</opErrstr>
></cliOutput>
>[kubeexec] ERROR 2018/06/08 09:19:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume stop vol_5db55c483f6984d9917cfe4b3c8b3cbc force] on glusterfs-storage-pg4xc: Err[command terminated with exit code 1]: Stdout []: Stderr [volume stop: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>]
>[cmdexec] ERROR 2018/06/08 09:19:51 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:144: Unable to stop volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume stop: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[kubeexec] ERROR 2018/06/08 09:19:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume delete vol_5db55c483f6984d9917cfe4b3c8b3cbc] on glusterfs-storage-pg4xc: Err[command terminated with exit code 1]: Stdout []: Stderr [volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>]
>[cmdexec] ERROR 2018/06/08 09:19:51 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:153: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[heketi] ERROR 2018/06/08 09:19:51 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:525: Unable to delete volume: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[heketi] ERROR 2018/06/08 09:19:51 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:429: Error executing delete volume: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[asynchttp] INFO 2018/06/08 09:19:51 asynchttp.go:292: Completed job a87500ed17aaa8058a96f89376065a33 in 763.735341ms
>[heketi] ERROR 2018/06/08 09:19:51 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:1185: Delete Volume Failed: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[negroni] Started GET /queue/a87500ed17aaa8058a96f89376065a33
>[negroni] Completed 500 Internal Server Error in 136.414Âµs
>[negroni] Started DELETE /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 202 Accepted in 36.092438ms
>[asynchttp] INFO 2018/06/08 09:20:05 asynchttp.go:288: Started job 64761d056f81ad1c415cd0ad05fd737f
>[heketi] INFO 2018/06/08 09:20:05 Started async operation: Delete Volume
>[negroni] Started GET /queue/64761d056f81ad1c415cd0ad05fd737f
>[negroni] Completed 200 OK in 108.592Âµs
>[kubeexec] DEBUG 2018/06/08 09:20:06 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script snapshot list vol_9c85de66c12db0a72b4d16fe888ff74d --xml
>Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
><cliOutput>
>  <opRet>-1</opRet>
>  <opErrno>30806</opErrno>
>  <opErrstr>Volume (vol_9c85de66c12db0a72b4d16fe888ff74d) does not exist</opErrstr>
></cliOutput>
>[kubeexec] ERROR 2018/06/08 09:20:06 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume stop vol_9c85de66c12db0a72b4d16fe888ff74d force] on glusterfs-storage-gxp7c: Err[command terminated with exit code 1]: Stdout []: Stderr [volume stop: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>]
>[cmdexec] ERROR 2018/06/08 09:20:06 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:144: Unable to stop volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume stop: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[kubeexec] ERROR 2018/06/08 09:20:06 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume delete vol_9c85de66c12db0a72b4d16fe888ff74d] on glusterfs-storage-gxp7c: Err[command terminated with exit code 1]: Stdout []: Stderr [volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>]
>[cmdexec] ERROR 2018/06/08 09:20:06 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:153: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[heketi] ERROR 2018/06/08 09:20:06 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:525: Unable to delete volume: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[heketi] ERROR 2018/06/08 09:20:06 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:429: Error executing delete volume: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[asynchttp] INFO 2018/06/08 09:20:06 asynchttp.go:292: Completed job 64761d056f81ad1c415cd0ad05fd737f in 734.563436ms
>[heketi] ERROR 2018/06/08 09:20:06 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:1185: Delete Volume Failed: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[negroni] Started GET /queue/64761d056f81ad1c415cd0ad05fd737f
>[negroni] Completed 500 Internal Server Error in 194.271Âµs
>[heketi] INFO 2018/06/08 09:21:14 Starting Node Health Status refresh
>[cmdexec] INFO 2018/06/08 09:21:14 Check Glusterd service status in node dhcp46-187.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 09:21:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:33 UTC; 2h 57min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48fed567_6ae4_11e8_bc4b_005056a5f18a.slice/docker-a506d9aaca4b440d184df40d01e90b2040811c8090bf4c19d892081307b28c0c.scope/system.slice/glusterd.service
>           ââ  433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1228 /usr/sbin/glusterfsd -s 10.70.46.187 --volfile-id heketidbstorage.10.70.46.187.var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.187-var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.pid -S /var/run/gluster/c08898c2e3b0190c62df3a9d4a013c92.socket --brick-name /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.log --xlator-option *-posix.glusterd-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ20535 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/8fb729a0bbd51b204491c603c84acaa4.socket --xlator-option *replicate*.node-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034
>
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Unit glusterd.service entered failed state.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: glusterd.service failed.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:33 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 09:21:14 Periodic health check status: node 278bd6b4e16a8e62ef15aaae22e6abc1 up=true
>[cmdexec] INFO 2018/06/08 09:21:14 Check Glusterd service status in node dhcp46-122.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 09:21:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:42 UTC; 2h 57min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4900c968_6ae4_11e8_bc4b_005056a5f18a.slice/docker-438fb5fb16a82a3e701b28fa1f1ef8623dc37ca06374c604e5781a2b2ab2e06d.scope/system.slice/glusterd.service
>           ââ  433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1106 /usr/sbin/glusterfsd -s 10.70.46.122 --volfile-id heketidbstorage.10.70.46.122.var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.122-var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.pid -S /var/run/gluster/9fa3ac3b12826912287adc88c26ac134.socket --brick-name /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_29e5b36d921cca1b0699871d9727c932/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.log --xlator-option *-posix.glusterd-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ21076 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f8994cdb09351d83e70da08261d99c64.socket --xlator-option *replicate*.node-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0
>
>Jun 08 06:23:40 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:42 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 09:21:14 Periodic health check status: node 70423cc0bcd044fe5ba8bbbd256a3e49 up=true
>[cmdexec] INFO 2018/06/08 09:21:14 Check Glusterd service status in node dhcp47-76.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 09:21:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 07:25:20 UTC; 1h 55min ago
>  Process: 828 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 830 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5cb2299_6aec_11e8_bc4b_005056a5f18a.slice/docker-7f9223abc9a8641cea68f374f4039235515978ee2dab970ac6cbef182c535426.scope/system.slice/glusterd.service
>           ââ  830 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1197 /usr/sbin/glusterfsd -s 10.70.47.76 --volfile-id heketidbstorage.10.70.47.76.var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.76-var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.pid -S /var/run/gluster/b6976f9ddc3abc90e1dfa91c35a7c611.socket --brick-name /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_54c06cdf52c6fe1e3a3f8a9d39cde33d/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.log --xlator-option *-posix.glusterd-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ13734 /usr/sbin/glusterfsd -s 10.70.47.76 --volfile-id vol_bf31af76ef6c54e7e8f24f4d8711cb22.10.70.47.76.var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_c6935061d1c9dbb05f316e3d87080a38-brick -p /var/run/gluster/vols/vol_bf31af76ef6c54e7e8f24f4d8711cb22/10.70.47.76-var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_c6935061d1c9dbb05f316e3d87080a38-brick.pid -S /var/run/gluster/7aabe8187d4b998d498dd5d4edbd2626.socket --brick-name /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_c6935061d1c9dbb05f316e3d87080a38/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_c6935061d1c9dbb05f316e3d87080a38-brick.log --xlator-option *-posix.glusterd-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94 --brick-port 49153 --xlator-option vol_bf31af76ef6c54e7e8f24f4d8711cb22-server.listen-port=49153
>           ââ18989 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/ec468453b11caed5259f5e1abcc65b44.socket --xlator-option *replicate*.node-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94
>
>Jun 08 07:25:17 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 07:25:20 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Jun 08 08:53:58 dhcp47-76.lab.eng.blr.redhat.com glusterd[830]: [2018-06-08 08:53:58.338441] C [rpc-clnt.c:465:rpc_clnt_fill_request_info] 0-management: cannot lookup the saved frame corresponding to xid (103)
>[heketi] INFO 2018/06/08 09:21:14 Periodic health check status: node d942fe6c0ee5691b7cc263968f97b650 up=true
>[heketi] INFO 2018/06/08 09:21:14 Cleaned 0 nodes from health cache
>[negroni] Started DELETE /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 202 Accepted in 17.042807ms
>[asynchttp] INFO 2018/06/08 09:22:05 asynchttp.go:288: Started job 19c38fef7ce5cca7827d4664e3c834d4
>[heketi] INFO 2018/06/08 09:22:05 Started async operation: Delete Volume
>[negroni] Started GET /queue/19c38fef7ce5cca7827d4664e3c834d4
>[negroni] Completed 200 OK in 187.572Âµs
>[kubeexec] DEBUG 2018/06/08 09:22:06 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script snapshot list vol_5db55c483f6984d9917cfe4b3c8b3cbc --xml
>Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
><cliOutput>
>  <opRet>-1</opRet>
>  <opErrno>30806</opErrno>
>  <opErrstr>Volume (vol_5db55c483f6984d9917cfe4b3c8b3cbc) does not exist</opErrstr>
></cliOutput>
>[kubeexec] ERROR 2018/06/08 09:22:06 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume stop vol_5db55c483f6984d9917cfe4b3c8b3cbc force] on glusterfs-storage-pg4xc: Err[command terminated with exit code 1]: Stdout []: Stderr [volume stop: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>]
>[cmdexec] ERROR 2018/06/08 09:22:06 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:144: Unable to stop volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume stop: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[kubeexec] ERROR 2018/06/08 09:22:06 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume delete vol_5db55c483f6984d9917cfe4b3c8b3cbc] on glusterfs-storage-pg4xc: Err[command terminated with exit code 1]: Stdout []: Stderr [volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>]
>[cmdexec] ERROR 2018/06/08 09:22:06 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:153: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[heketi] ERROR 2018/06/08 09:22:06 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:525: Unable to delete volume: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[heketi] ERROR 2018/06/08 09:22:06 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:429: Error executing delete volume: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[heketi] ERROR 2018/06/08 09:22:06 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:1185: Delete Volume Failed: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[asynchttp] INFO 2018/06/08 09:22:06 asynchttp.go:292: Completed job 19c38fef7ce5cca7827d4664e3c834d4 in 791.373785ms
>[negroni] Started GET /queue/19c38fef7ce5cca7827d4664e3c834d4
>[negroni] Completed 500 Internal Server Error in 312.924Âµs
>[negroni] Started DELETE /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 202 Accepted in 33.337412ms
>[asynchttp] INFO 2018/06/08 09:22:20 asynchttp.go:288: Started job 01c49a4f8b40bb2ce11f2efce29cff1c
>[heketi] INFO 2018/06/08 09:22:20 Started async operation: Delete Volume
>[negroni] Started GET /queue/01c49a4f8b40bb2ce11f2efce29cff1c
>[negroni] Completed 200 OK in 178.872Âµs
>[kubeexec] DEBUG 2018/06/08 09:22:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script snapshot list vol_9c85de66c12db0a72b4d16fe888ff74d --xml
>Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
><cliOutput>
>  <opRet>-1</opRet>
>  <opErrno>30806</opErrno>
>  <opErrstr>Volume (vol_9c85de66c12db0a72b4d16fe888ff74d) does not exist</opErrstr>
></cliOutput>
>[kubeexec] ERROR 2018/06/08 09:22:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume stop vol_9c85de66c12db0a72b4d16fe888ff74d force] on glusterfs-storage-gxp7c: Err[command terminated with exit code 1]: Stdout []: Stderr [volume stop: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>]
>[cmdexec] ERROR 2018/06/08 09:22:21 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:144: Unable to stop volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume stop: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[kubeexec] ERROR 2018/06/08 09:22:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume delete vol_9c85de66c12db0a72b4d16fe888ff74d] on glusterfs-storage-gxp7c: Err[command terminated with exit code 1]: Stdout []: Stderr [volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>]
>[cmdexec] ERROR 2018/06/08 09:22:21 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:153: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[heketi] ERROR 2018/06/08 09:22:21 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:525: Unable to delete volume: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[heketi] ERROR 2018/06/08 09:22:21 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:429: Error executing delete volume: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[heketi] ERROR 2018/06/08 09:22:21 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:1185: Delete Volume Failed: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[asynchttp] INFO 2018/06/08 09:22:21 asynchttp.go:292: Completed job 01c49a4f8b40bb2ce11f2efce29cff1c in 762.286487ms
>[negroni] Started GET /queue/01c49a4f8b40bb2ce11f2efce29cff1c
>[negroni] Completed 500 Internal Server Error in 200.842Âµs
>[heketi] INFO 2018/06/08 09:23:14 Starting Node Health Status refresh
>[cmdexec] INFO 2018/06/08 09:23:14 Check Glusterd service status in node dhcp46-187.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 09:23:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:33 UTC; 2h 59min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48fed567_6ae4_11e8_bc4b_005056a5f18a.slice/docker-a506d9aaca4b440d184df40d01e90b2040811c8090bf4c19d892081307b28c0c.scope/system.slice/glusterd.service
>           ââ  433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1228 /usr/sbin/glusterfsd -s 10.70.46.187 --volfile-id heketidbstorage.10.70.46.187.var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.187-var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.pid -S /var/run/gluster/c08898c2e3b0190c62df3a9d4a013c92.socket --brick-name /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.log --xlator-option *-posix.glusterd-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ20535 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/8fb729a0bbd51b204491c603c84acaa4.socket --xlator-option *replicate*.node-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034
>
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Unit glusterd.service entered failed state.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: glusterd.service failed.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:33 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 09:23:14 Periodic health check status: node 278bd6b4e16a8e62ef15aaae22e6abc1 up=true
>[cmdexec] INFO 2018/06/08 09:23:14 Check Glusterd service status in node dhcp46-122.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 09:23:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:42 UTC; 2h 59min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4900c968_6ae4_11e8_bc4b_005056a5f18a.slice/docker-438fb5fb16a82a3e701b28fa1f1ef8623dc37ca06374c604e5781a2b2ab2e06d.scope/system.slice/glusterd.service
>           ââ  433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1106 /usr/sbin/glusterfsd -s 10.70.46.122 --volfile-id heketidbstorage.10.70.46.122.var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.122-var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.pid -S /var/run/gluster/9fa3ac3b12826912287adc88c26ac134.socket --brick-name /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_29e5b36d921cca1b0699871d9727c932/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.log --xlator-option *-posix.glusterd-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ21076 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f8994cdb09351d83e70da08261d99c64.socket --xlator-option *replicate*.node-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0
>
>Jun 08 06:23:40 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:42 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 09:23:14 Periodic health check status: node 70423cc0bcd044fe5ba8bbbd256a3e49 up=true
>[cmdexec] INFO 2018/06/08 09:23:14 Check Glusterd service status in node dhcp47-76.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 09:23:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 07:25:20 UTC; 1h 57min ago
>  Process: 828 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 830 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5cb2299_6aec_11e8_bc4b_005056a5f18a.slice/docker-7f9223abc9a8641cea68f374f4039235515978ee2dab970ac6cbef182c535426.scope/system.slice/glusterd.service
>           ââ  830 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1197 /usr/sbin/glusterfsd -s 10.70.47.76 --volfile-id heketidbstorage.10.70.47.76.var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.76-var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.pid -S /var/run/gluster/b6976f9ddc3abc90e1dfa91c35a7c611.socket --brick-name /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_54c06cdf52c6fe1e3a3f8a9d39cde33d/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.log --xlator-option *-posix.glusterd-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ13734 /usr/sbin/glusterfsd -s 10.70.47.76 --volfile-id vol_bf31af76ef6c54e7e8f24f4d8711cb22.10.70.47.76.var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_c6935061d1c9dbb05f316e3d87080a38-brick -p /var/run/gluster/vols/vol_bf31af76ef6c54e7e8f24f4d8711cb22/10.70.47.76-var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_c6935061d1c9dbb05f316e3d87080a38-brick.pid -S /var/run/gluster/7aabe8187d4b998d498dd5d4edbd2626.socket --brick-name /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_c6935061d1c9dbb05f316e3d87080a38/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_c6935061d1c9dbb05f316e3d87080a38-brick.log --xlator-option *-posix.glusterd-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94 --brick-port 49153 --xlator-option vol_bf31af76ef6c54e7e8f24f4d8711cb22-server.listen-port=49153
>           ââ18989 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/ec468453b11caed5259f5e1abcc65b44.socket --xlator-option *replicate*.node-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94
>
>Jun 08 07:25:17 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 07:25:20 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Jun 08 08:53:58 dhcp47-76.lab.eng.blr.redhat.com glusterd[830]: [2018-06-08 08:53:58.338441] C [rpc-clnt.c:465:rpc_clnt_fill_request_info] 0-management: cannot lookup the saved frame corresponding to xid (103)
>[heketi] INFO 2018/06/08 09:23:14 Periodic health check status: node d942fe6c0ee5691b7cc263968f97b650 up=true
>[heketi] INFO 2018/06/08 09:23:14 Cleaned 0 nodes from health cache
>[negroni] Started DELETE /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 202 Accepted in 21.866363ms
>[asynchttp] INFO 2018/06/08 09:24:20 asynchttp.go:288: Started job f41582fe34d6cdebdee5f904c64d6207
>[heketi] INFO 2018/06/08 09:24:20 Started async operation: Delete Volume
>[negroni] Started GET /queue/f41582fe34d6cdebdee5f904c64d6207
>[negroni] Completed 200 OK in 121.01Âµs
>[kubeexec] DEBUG 2018/06/08 09:24:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script snapshot list vol_5db55c483f6984d9917cfe4b3c8b3cbc --xml
>Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
><cliOutput>
>  <opRet>-1</opRet>
>  <opErrno>30806</opErrno>
>  <opErrstr>Volume (vol_5db55c483f6984d9917cfe4b3c8b3cbc) does not exist</opErrstr>
></cliOutput>
>[kubeexec] ERROR 2018/06/08 09:24:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume stop vol_5db55c483f6984d9917cfe4b3c8b3cbc force] on glusterfs-storage-pg4xc: Err[command terminated with exit code 1]: Stdout []: Stderr [volume stop: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>]
>[cmdexec] ERROR 2018/06/08 09:24:21 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:144: Unable to stop volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume stop: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[kubeexec] ERROR 2018/06/08 09:24:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume delete vol_5db55c483f6984d9917cfe4b3c8b3cbc] on glusterfs-storage-pg4xc: Err[command terminated with exit code 1]: Stdout []: Stderr [volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>]
>[cmdexec] ERROR 2018/06/08 09:24:21 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:153: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[heketi] ERROR 2018/06/08 09:24:21 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:525: Unable to delete volume: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[heketi] ERROR 2018/06/08 09:24:21 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:429: Error executing delete volume: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[asynchttp] INFO 2018/06/08 09:24:21 asynchttp.go:292: Completed job f41582fe34d6cdebdee5f904c64d6207 in 806.748405ms
>[heketi] ERROR 2018/06/08 09:24:21 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:1185: Delete Volume Failed: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[negroni] Started GET /queue/f41582fe34d6cdebdee5f904c64d6207
>[negroni] Completed 500 Internal Server Error in 255.449Âµs
>[negroni] Started DELETE /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 202 Accepted in 27.973522ms
>[asynchttp] INFO 2018/06/08 09:24:35 asynchttp.go:288: Started job a4cd993cad9645319aff2135f818a8fc
>[heketi] INFO 2018/06/08 09:24:35 Started async operation: Delete Volume
>[negroni] Started GET /queue/a4cd993cad9645319aff2135f818a8fc
>[negroni] Completed 200 OK in 212.125Âµs
>[kubeexec] DEBUG 2018/06/08 09:24:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script snapshot list vol_9c85de66c12db0a72b4d16fe888ff74d --xml
>Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
><cliOutput>
>  <opRet>-1</opRet>
>  <opErrno>30806</opErrno>
>  <opErrstr>Volume (vol_9c85de66c12db0a72b4d16fe888ff74d) does not exist</opErrstr>
></cliOutput>
>[kubeexec] ERROR 2018/06/08 09:24:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume stop vol_9c85de66c12db0a72b4d16fe888ff74d force] on glusterfs-storage-gxp7c: Err[command terminated with exit code 1]: Stdout []: Stderr [volume stop: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>]
>[cmdexec] ERROR 2018/06/08 09:24:36 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:144: Unable to stop volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume stop: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[kubeexec] ERROR 2018/06/08 09:24:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume delete vol_9c85de66c12db0a72b4d16fe888ff74d] on glusterfs-storage-gxp7c: Err[command terminated with exit code 1]: Stdout []: Stderr [volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>]
>[cmdexec] ERROR 2018/06/08 09:24:36 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:153: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[heketi] ERROR 2018/06/08 09:24:36 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:525: Unable to delete volume: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[heketi] ERROR 2018/06/08 09:24:36 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:429: Error executing delete volume: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[asynchttp] INFO 2018/06/08 09:24:36 asynchttp.go:292: Completed job a4cd993cad9645319aff2135f818a8fc in 775.990664ms
>[heketi] ERROR 2018/06/08 09:24:36 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:1185: Delete Volume Failed: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[negroni] Started GET /queue/a4cd993cad9645319aff2135f818a8fc
>[negroni] Completed 500 Internal Server Error in 195.298Âµs
>[heketi] INFO 2018/06/08 09:25:14 Starting Node Health Status refresh
>[cmdexec] INFO 2018/06/08 09:25:14 Check Glusterd service status in node dhcp46-187.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 09:25:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:33 UTC; 3h 1min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48fed567_6ae4_11e8_bc4b_005056a5f18a.slice/docker-a506d9aaca4b440d184df40d01e90b2040811c8090bf4c19d892081307b28c0c.scope/system.slice/glusterd.service
>           ââ  433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1228 /usr/sbin/glusterfsd -s 10.70.46.187 --volfile-id heketidbstorage.10.70.46.187.var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.187-var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.pid -S /var/run/gluster/c08898c2e3b0190c62df3a9d4a013c92.socket --brick-name /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.log --xlator-option *-posix.glusterd-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ20535 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/8fb729a0bbd51b204491c603c84acaa4.socket --xlator-option *replicate*.node-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034
>
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Unit glusterd.service entered failed state.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: glusterd.service failed.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:33 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 09:25:14 Periodic health check status: node 278bd6b4e16a8e62ef15aaae22e6abc1 up=true
>[cmdexec] INFO 2018/06/08 09:25:14 Check Glusterd service status in node dhcp46-122.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 09:25:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:42 UTC; 3h 1min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4900c968_6ae4_11e8_bc4b_005056a5f18a.slice/docker-438fb5fb16a82a3e701b28fa1f1ef8623dc37ca06374c604e5781a2b2ab2e06d.scope/system.slice/glusterd.service
>           ââ  433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1106 /usr/sbin/glusterfsd -s 10.70.46.122 --volfile-id heketidbstorage.10.70.46.122.var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.122-var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.pid -S /var/run/gluster/9fa3ac3b12826912287adc88c26ac134.socket --brick-name /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_29e5b36d921cca1b0699871d9727c932/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.log --xlator-option *-posix.glusterd-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ21076 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f8994cdb09351d83e70da08261d99c64.socket --xlator-option *replicate*.node-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0
>
>Jun 08 06:23:40 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:42 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 09:25:14 Periodic health check status: node 70423cc0bcd044fe5ba8bbbd256a3e49 up=true
>[cmdexec] INFO 2018/06/08 09:25:14 Check Glusterd service status in node dhcp47-76.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 09:25:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 07:25:20 UTC; 1h 59min ago
>  Process: 828 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 830 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5cb2299_6aec_11e8_bc4b_005056a5f18a.slice/docker-7f9223abc9a8641cea68f374f4039235515978ee2dab970ac6cbef182c535426.scope/system.slice/glusterd.service
>           ââ  830 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1197 /usr/sbin/glusterfsd -s 10.70.47.76 --volfile-id heketidbstorage.10.70.47.76.var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.76-var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.pid -S /var/run/gluster/b6976f9ddc3abc90e1dfa91c35a7c611.socket --brick-name /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_54c06cdf52c6fe1e3a3f8a9d39cde33d/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.log --xlator-option *-posix.glusterd-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ13734 /usr/sbin/glusterfsd -s 10.70.47.76 --volfile-id vol_bf31af76ef6c54e7e8f24f4d8711cb22.10.70.47.76.var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_c6935061d1c9dbb05f316e3d87080a38-brick -p /var/run/gluster/vols/vol_bf31af76ef6c54e7e8f24f4d8711cb22/10.70.47.76-var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_c6935061d1c9dbb05f316e3d87080a38-brick.pid -S /var/run/gluster/7aabe8187d4b998d498dd5d4edbd2626.socket --brick-name /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_c6935061d1c9dbb05f316e3d87080a38/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_c6935061d1c9dbb05f316e3d87080a38-brick.log --xlator-option *-posix.glusterd-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94 --brick-port 49153 --xlator-option vol_bf31af76ef6c54e7e8f24f4d8711cb22-server.listen-port=49153
>           ââ18989 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/ec468453b11caed5259f5e1abcc65b44.socket --xlator-option *replicate*.node-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94
>
>Jun 08 07:25:17 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 07:25:20 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Jun 08 08:53:58 dhcp47-76.lab.eng.blr.redhat.com glusterd[830]: [2018-06-08 08:53:58.338441] C [rpc-clnt.c:465:rpc_clnt_fill_request_info] 0-management: cannot lookup the saved frame corresponding to xid (103)
>[heketi] INFO 2018/06/08 09:25:14 Periodic health check status: node d942fe6c0ee5691b7cc263968f97b650 up=true
>[heketi] INFO 2018/06/08 09:25:14 Cleaned 0 nodes from health cache
>[negroni] Started DELETE /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 202 Accepted in 30.343578ms
>[asynchttp] INFO 2018/06/08 09:26:35 asynchttp.go:288: Started job 03af57740675b5e7d8d6372fdbff500e
>[heketi] INFO 2018/06/08 09:26:35 Started async operation: Delete Volume
>[negroni] Started GET /queue/03af57740675b5e7d8d6372fdbff500e
>[negroni] Completed 200 OK in 181.205Âµs
>[kubeexec] DEBUG 2018/06/08 09:26:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script snapshot list vol_5db55c483f6984d9917cfe4b3c8b3cbc --xml
>Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
><cliOutput>
>  <opRet>-1</opRet>
>  <opErrno>30806</opErrno>
>  <opErrstr>Volume (vol_5db55c483f6984d9917cfe4b3c8b3cbc) does not exist</opErrstr>
></cliOutput>
>[kubeexec] ERROR 2018/06/08 09:26:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume stop vol_5db55c483f6984d9917cfe4b3c8b3cbc force] on glusterfs-storage-pg4xc: Err[command terminated with exit code 1]: Stdout []: Stderr [volume stop: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>]
>[cmdexec] ERROR 2018/06/08 09:26:36 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:144: Unable to stop volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume stop: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[kubeexec] ERROR 2018/06/08 09:26:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume delete vol_5db55c483f6984d9917cfe4b3c8b3cbc] on glusterfs-storage-pg4xc: Err[command terminated with exit code 1]: Stdout []: Stderr [volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>]
>[cmdexec] ERROR 2018/06/08 09:26:36 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:153: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[heketi] ERROR 2018/06/08 09:26:36 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:525: Unable to delete volume: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[heketi] ERROR 2018/06/08 09:26:36 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:429: Error executing delete volume: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[heketi] ERROR 2018/06/08 09:26:36 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:1185: Delete Volume Failed: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[asynchttp] INFO 2018/06/08 09:26:36 asynchttp.go:292: Completed job 03af57740675b5e7d8d6372fdbff500e in 790.272346ms
>[negroni] Started GET /queue/03af57740675b5e7d8d6372fdbff500e
>[negroni] Completed 500 Internal Server Error in 190.478Âµs
>[negroni] Started DELETE /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 202 Accepted in 21.589174ms
>[asynchttp] INFO 2018/06/08 09:26:50 asynchttp.go:288: Started job cc678afc81636b0b6c9d6d0381163441
>[heketi] INFO 2018/06/08 09:26:50 Started async operation: Delete Volume
>[negroni] Started GET /queue/cc678afc81636b0b6c9d6d0381163441
>[negroni] Completed 200 OK in 185.029Âµs
>[kubeexec] DEBUG 2018/06/08 09:26:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script snapshot list vol_9c85de66c12db0a72b4d16fe888ff74d --xml
>Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
><cliOutput>
>  <opRet>-1</opRet>
>  <opErrno>30806</opErrno>
>  <opErrstr>Volume (vol_9c85de66c12db0a72b4d16fe888ff74d) does not exist</opErrstr>
></cliOutput>
>[kubeexec] ERROR 2018/06/08 09:26:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume stop vol_9c85de66c12db0a72b4d16fe888ff74d force] on glusterfs-storage-gxp7c: Err[command terminated with exit code 1]: Stdout []: Stderr [volume stop: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>]
>[cmdexec] ERROR 2018/06/08 09:26:51 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:144: Unable to stop volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume stop: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[kubeexec] ERROR 2018/06/08 09:26:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume delete vol_9c85de66c12db0a72b4d16fe888ff74d] on glusterfs-storage-gxp7c: Err[command terminated with exit code 1]: Stdout []: Stderr [volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>]
>[cmdexec] ERROR 2018/06/08 09:26:51 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:153: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[heketi] ERROR 2018/06/08 09:26:51 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:525: Unable to delete volume: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[heketi] ERROR 2018/06/08 09:26:51 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:429: Error executing delete volume: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[heketi] ERROR 2018/06/08 09:26:51 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:1185: Delete Volume Failed: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[asynchttp] INFO 2018/06/08 09:26:51 asynchttp.go:292: Completed job cc678afc81636b0b6c9d6d0381163441 in 756.897861ms
>[negroni] Started GET /queue/cc678afc81636b0b6c9d6d0381163441
>[negroni] Completed 500 Internal Server Error in 291.308Âµs
>[heketi] INFO 2018/06/08 09:27:14 Starting Node Health Status refresh
>[cmdexec] INFO 2018/06/08 09:27:14 Check Glusterd service status in node dhcp46-187.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 09:27:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:33 UTC; 3h 3min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48fed567_6ae4_11e8_bc4b_005056a5f18a.slice/docker-a506d9aaca4b440d184df40d01e90b2040811c8090bf4c19d892081307b28c0c.scope/system.slice/glusterd.service
>           ââ  433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1228 /usr/sbin/glusterfsd -s 10.70.46.187 --volfile-id heketidbstorage.10.70.46.187.var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.187-var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.pid -S /var/run/gluster/c08898c2e3b0190c62df3a9d4a013c92.socket --brick-name /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.log --xlator-option *-posix.glusterd-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ20535 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/8fb729a0bbd51b204491c603c84acaa4.socket --xlator-option *replicate*.node-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034
>
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Unit glusterd.service entered failed state.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: glusterd.service failed.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:33 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 09:27:14 Periodic health check status: node 278bd6b4e16a8e62ef15aaae22e6abc1 up=true
>[cmdexec] INFO 2018/06/08 09:27:14 Check Glusterd service status in node dhcp46-122.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 09:27:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:42 UTC; 3h 3min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4900c968_6ae4_11e8_bc4b_005056a5f18a.slice/docker-438fb5fb16a82a3e701b28fa1f1ef8623dc37ca06374c604e5781a2b2ab2e06d.scope/system.slice/glusterd.service
>           ââ  433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1106 /usr/sbin/glusterfsd -s 10.70.46.122 --volfile-id heketidbstorage.10.70.46.122.var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.122-var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.pid -S /var/run/gluster/9fa3ac3b12826912287adc88c26ac134.socket --brick-name /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_29e5b36d921cca1b0699871d9727c932/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.log --xlator-option *-posix.glusterd-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ21076 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f8994cdb09351d83e70da08261d99c64.socket --xlator-option *replicate*.node-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0
>
>Jun 08 06:23:40 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:42 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 09:27:14 Periodic health check status: node 70423cc0bcd044fe5ba8bbbd256a3e49 up=true
>[cmdexec] INFO 2018/06/08 09:27:14 Check Glusterd service status in node dhcp47-76.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 09:27:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 07:25:20 UTC; 2h 1min ago
>  Process: 828 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 830 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5cb2299_6aec_11e8_bc4b_005056a5f18a.slice/docker-7f9223abc9a8641cea68f374f4039235515978ee2dab970ac6cbef182c535426.scope/system.slice/glusterd.service
>           ââ  830 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1197 /usr/sbin/glusterfsd -s 10.70.47.76 --volfile-id heketidbstorage.10.70.47.76.var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.76-var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.pid -S /var/run/gluster/b6976f9ddc3abc90e1dfa91c35a7c611.socket --brick-name /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_54c06cdf52c6fe1e3a3f8a9d39cde33d/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.log --xlator-option *-posix.glusterd-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ13734 /usr/sbin/glusterfsd -s 10.70.47.76 --volfile-id vol_bf31af76ef6c54e7e8f24f4d8711cb22.10.70.47.76.var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_c6935061d1c9dbb05f316e3d87080a38-brick -p /var/run/gluster/vols/vol_bf31af76ef6c54e7e8f24f4d8711cb22/10.70.47.76-var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_c6935061d1c9dbb05f316e3d87080a38-brick.pid -S /var/run/gluster/7aabe8187d4b998d498dd5d4edbd2626.socket --brick-name /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_c6935061d1c9dbb05f316e3d87080a38/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_c6935061d1c9dbb05f316e3d87080a38-brick.log --xlator-option *-posix.glusterd-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94 --brick-port 49153 --xlator-option vol_bf31af76ef6c54e7e8f24f4d8711cb22-server.listen-port=49153
>           ââ18989 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/ec468453b11caed5259f5e1abcc65b44.socket --xlator-option *replicate*.node-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94
>
>Jun 08 07:25:17 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 07:25:20 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Jun 08 08:53:58 dhcp47-76.lab.eng.blr.redhat.com glusterd[830]: [2018-06-08 08:53:58.338441] C [rpc-clnt.c:465:rpc_clnt_fill_request_info] 0-management: cannot lookup the saved frame corresponding to xid (103)
>[heketi] INFO 2018/06/08 09:27:14 Periodic health check status: node d942fe6c0ee5691b7cc263968f97b650 up=true
>[heketi] INFO 2018/06/08 09:27:14 Cleaned 0 nodes from health cache
>[negroni] Started DELETE /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 202 Accepted in 17.496768ms
>[asynchttp] INFO 2018/06/08 09:28:50 asynchttp.go:288: Started job 8e9d7af5d01821c5c89264a81e1b5e2c
>[heketi] INFO 2018/06/08 09:28:50 Started async operation: Delete Volume
>[negroni] Started GET /queue/8e9d7af5d01821c5c89264a81e1b5e2c
>[negroni] Completed 200 OK in 174.149Âµs
>[kubeexec] DEBUG 2018/06/08 09:28:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script snapshot list vol_5db55c483f6984d9917cfe4b3c8b3cbc --xml
>Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
><cliOutput>
>  <opRet>-1</opRet>
>  <opErrno>30806</opErrno>
>  <opErrstr>Volume (vol_5db55c483f6984d9917cfe4b3c8b3cbc) does not exist</opErrstr>
></cliOutput>
>[kubeexec] ERROR 2018/06/08 09:28:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume stop vol_5db55c483f6984d9917cfe4b3c8b3cbc force] on glusterfs-storage-pg4xc: Err[command terminated with exit code 1]: Stdout []: Stderr [volume stop: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>]
>[cmdexec] ERROR 2018/06/08 09:28:51 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:144: Unable to stop volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume stop: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[kubeexec] ERROR 2018/06/08 09:28:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume delete vol_5db55c483f6984d9917cfe4b3c8b3cbc] on glusterfs-storage-pg4xc: Err[command terminated with exit code 1]: Stdout []: Stderr [volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>]
>[cmdexec] ERROR 2018/06/08 09:28:51 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:153: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[heketi] ERROR 2018/06/08 09:28:51 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:525: Unable to delete volume: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[heketi] ERROR 2018/06/08 09:28:51 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:429: Error executing delete volume: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[heketi] ERROR 2018/06/08 09:28:51 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:1185: Delete Volume Failed: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[asynchttp] INFO 2018/06/08 09:28:51 asynchttp.go:292: Completed job 8e9d7af5d01821c5c89264a81e1b5e2c in 794.826895ms
>[negroni] Started GET /queue/8e9d7af5d01821c5c89264a81e1b5e2c
>[negroni] Completed 500 Internal Server Error in 315.014Âµs
>[negroni] Started DELETE /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 202 Accepted in 30.337823ms
>[asynchttp] INFO 2018/06/08 09:29:05 asynchttp.go:288: Started job a599a3a23058856d84483803f3353abe
>[heketi] INFO 2018/06/08 09:29:05 Started async operation: Delete Volume
>[negroni] Started GET /queue/a599a3a23058856d84483803f3353abe
>[negroni] Completed 200 OK in 262.228Âµs
>[kubeexec] DEBUG 2018/06/08 09:29:06 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script snapshot list vol_9c85de66c12db0a72b4d16fe888ff74d --xml
>Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
><cliOutput>
>  <opRet>-1</opRet>
>  <opErrno>30806</opErrno>
>  <opErrstr>Volume (vol_9c85de66c12db0a72b4d16fe888ff74d) does not exist</opErrstr>
></cliOutput>
>[kubeexec] ERROR 2018/06/08 09:29:06 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume stop vol_9c85de66c12db0a72b4d16fe888ff74d force] on glusterfs-storage-gxp7c: Err[command terminated with exit code 1]: Stdout []: Stderr [volume stop: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>]
>[cmdexec] ERROR 2018/06/08 09:29:06 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:144: Unable to stop volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume stop: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[kubeexec] ERROR 2018/06/08 09:29:06 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume delete vol_9c85de66c12db0a72b4d16fe888ff74d] on glusterfs-storage-gxp7c: Err[command terminated with exit code 1]: Stdout []: Stderr [volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>]
>[cmdexec] ERROR 2018/06/08 09:29:06 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:153: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[heketi] ERROR 2018/06/08 09:29:06 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:525: Unable to delete volume: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[heketi] ERROR 2018/06/08 09:29:06 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:429: Error executing delete volume: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[heketi] ERROR 2018/06/08 09:29:06 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:1185: Delete Volume Failed: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[asynchttp] INFO 2018/06/08 09:29:06 asynchttp.go:292: Completed job a599a3a23058856d84483803f3353abe in 751.08926ms
>[negroni] Started GET /queue/a599a3a23058856d84483803f3353abe
>[negroni] Completed 500 Internal Server Error in 278.173Âµs
>[heketi] INFO 2018/06/08 09:29:14 Starting Node Health Status refresh
>[cmdexec] INFO 2018/06/08 09:29:14 Check Glusterd service status in node dhcp46-187.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 09:29:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:33 UTC; 3h 5min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48fed567_6ae4_11e8_bc4b_005056a5f18a.slice/docker-a506d9aaca4b440d184df40d01e90b2040811c8090bf4c19d892081307b28c0c.scope/system.slice/glusterd.service
>           ââ  433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1228 /usr/sbin/glusterfsd -s 10.70.46.187 --volfile-id heketidbstorage.10.70.46.187.var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.187-var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.pid -S /var/run/gluster/c08898c2e3b0190c62df3a9d4a013c92.socket --brick-name /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.log --xlator-option *-posix.glusterd-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ20535 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/8fb729a0bbd51b204491c603c84acaa4.socket --xlator-option *replicate*.node-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034
>
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Unit glusterd.service entered failed state.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: glusterd.service failed.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:33 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 09:29:14 Periodic health check status: node 278bd6b4e16a8e62ef15aaae22e6abc1 up=true
>[cmdexec] INFO 2018/06/08 09:29:14 Check Glusterd service status in node dhcp46-122.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 09:29:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:42 UTC; 3h 5min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4900c968_6ae4_11e8_bc4b_005056a5f18a.slice/docker-438fb5fb16a82a3e701b28fa1f1ef8623dc37ca06374c604e5781a2b2ab2e06d.scope/system.slice/glusterd.service
>           ââ  433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1106 /usr/sbin/glusterfsd -s 10.70.46.122 --volfile-id heketidbstorage.10.70.46.122.var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.122-var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.pid -S /var/run/gluster/9fa3ac3b12826912287adc88c26ac134.socket --brick-name /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_29e5b36d921cca1b0699871d9727c932/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.log --xlator-option *-posix.glusterd-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ21076 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f8994cdb09351d83e70da08261d99c64.socket --xlator-option *replicate*.node-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0
>
>Jun 08 06:23:40 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:42 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 09:29:14 Periodic health check status: node 70423cc0bcd044fe5ba8bbbd256a3e49 up=true
>[cmdexec] INFO 2018/06/08 09:29:14 Check Glusterd service status in node dhcp47-76.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 09:29:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 07:25:20 UTC; 2h 3min ago
>  Process: 828 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 830 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5cb2299_6aec_11e8_bc4b_005056a5f18a.slice/docker-7f9223abc9a8641cea68f374f4039235515978ee2dab970ac6cbef182c535426.scope/system.slice/glusterd.service
>           ââ  830 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1197 /usr/sbin/glusterfsd -s 10.70.47.76 --volfile-id heketidbstorage.10.70.47.76.var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.76-var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.pid -S /var/run/gluster/b6976f9ddc3abc90e1dfa91c35a7c611.socket --brick-name /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_54c06cdf52c6fe1e3a3f8a9d39cde33d/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.log --xlator-option *-posix.glusterd-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ13734 /usr/sbin/glusterfsd -s 10.70.47.76 --volfile-id vol_bf31af76ef6c54e7e8f24f4d8711cb22.10.70.47.76.var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_c6935061d1c9dbb05f316e3d87080a38-brick -p /var/run/gluster/vols/vol_bf31af76ef6c54e7e8f24f4d8711cb22/10.70.47.76-var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_c6935061d1c9dbb05f316e3d87080a38-brick.pid -S /var/run/gluster/7aabe8187d4b998d498dd5d4edbd2626.socket --brick-name /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_c6935061d1c9dbb05f316e3d87080a38/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_c6935061d1c9dbb05f316e3d87080a38-brick.log --xlator-option *-posix.glusterd-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94 --brick-port 49153 --xlator-option vol_bf31af76ef6c54e7e8f24f4d8711cb22-server.listen-port=49153
>           ââ18989 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/ec468453b11caed5259f5e1abcc65b44.socket --xlator-option *replicate*.node-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94
>
>Jun 08 07:25:17 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 07:25:20 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Jun 08 08:53:58 dhcp47-76.lab.eng.blr.redhat.com glusterd[830]: [2018-06-08 08:53:58.338441] C [rpc-clnt.c:465:rpc_clnt_fill_request_info] 0-management: cannot lookup the saved frame corresponding to xid (103)
>[heketi] INFO 2018/06/08 09:29:14 Periodic health check status: node d942fe6c0ee5691b7cc263968f97b650 up=true
>[heketi] INFO 2018/06/08 09:29:14 Cleaned 0 nodes from health cache
>[negroni] Started DELETE /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 202 Accepted in 23.437629ms
>[asynchttp] INFO 2018/06/08 09:31:05 asynchttp.go:288: Started job 8ab0e7402a67844eb8d72184bdd9a568
>[heketi] INFO 2018/06/08 09:31:05 Started async operation: Delete Volume
>[negroni] Started GET /queue/8ab0e7402a67844eb8d72184bdd9a568
>[negroni] Completed 200 OK in 200.984Âµs
>[kubeexec] DEBUG 2018/06/08 09:31:06 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script snapshot list vol_5db55c483f6984d9917cfe4b3c8b3cbc --xml
>Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
><cliOutput>
>  <opRet>-1</opRet>
>  <opErrno>30806</opErrno>
>  <opErrstr>Volume (vol_5db55c483f6984d9917cfe4b3c8b3cbc) does not exist</opErrstr>
></cliOutput>
>[kubeexec] ERROR 2018/06/08 09:31:06 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume stop vol_5db55c483f6984d9917cfe4b3c8b3cbc force] on glusterfs-storage-pg4xc: Err[command terminated with exit code 1]: Stdout []: Stderr [volume stop: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>]
>[cmdexec] ERROR 2018/06/08 09:31:06 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:144: Unable to stop volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume stop: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[kubeexec] ERROR 2018/06/08 09:31:06 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume delete vol_5db55c483f6984d9917cfe4b3c8b3cbc] on glusterfs-storage-pg4xc: Err[command terminated with exit code 1]: Stdout []: Stderr [volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>]
>[cmdexec] ERROR 2018/06/08 09:31:06 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:153: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[heketi] ERROR 2018/06/08 09:31:06 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:525: Unable to delete volume: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[heketi] ERROR 2018/06/08 09:31:06 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:429: Error executing delete volume: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[heketi] ERROR 2018/06/08 09:31:06 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:1185: Delete Volume Failed: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[asynchttp] INFO 2018/06/08 09:31:06 asynchttp.go:292: Completed job 8ab0e7402a67844eb8d72184bdd9a568 in 786.928306ms
>[negroni] Started GET /queue/8ab0e7402a67844eb8d72184bdd9a568
>[negroni] Completed 500 Internal Server Error in 206.658Âµs
>[heketi] INFO 2018/06/08 09:31:14 Starting Node Health Status refresh
>[cmdexec] INFO 2018/06/08 09:31:14 Check Glusterd service status in node dhcp46-187.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 09:31:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:33 UTC; 3h 7min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48fed567_6ae4_11e8_bc4b_005056a5f18a.slice/docker-a506d9aaca4b440d184df40d01e90b2040811c8090bf4c19d892081307b28c0c.scope/system.slice/glusterd.service
>           ââ  433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1228 /usr/sbin/glusterfsd -s 10.70.46.187 --volfile-id heketidbstorage.10.70.46.187.var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.187-var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.pid -S /var/run/gluster/c08898c2e3b0190c62df3a9d4a013c92.socket --brick-name /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.log --xlator-option *-posix.glusterd-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ20535 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/8fb729a0bbd51b204491c603c84acaa4.socket --xlator-option *replicate*.node-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034
>
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Unit glusterd.service entered failed state.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: glusterd.service failed.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:33 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 09:31:14 Periodic health check status: node 278bd6b4e16a8e62ef15aaae22e6abc1 up=true
>[cmdexec] INFO 2018/06/08 09:31:14 Check Glusterd service status in node dhcp46-122.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 09:31:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:42 UTC; 3h 7min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4900c968_6ae4_11e8_bc4b_005056a5f18a.slice/docker-438fb5fb16a82a3e701b28fa1f1ef8623dc37ca06374c604e5781a2b2ab2e06d.scope/system.slice/glusterd.service
>           ââ  433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1106 /usr/sbin/glusterfsd -s 10.70.46.122 --volfile-id heketidbstorage.10.70.46.122.var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.122-var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.pid -S /var/run/gluster/9fa3ac3b12826912287adc88c26ac134.socket --brick-name /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_29e5b36d921cca1b0699871d9727c932/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.log --xlator-option *-posix.glusterd-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ21076 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f8994cdb09351d83e70da08261d99c64.socket --xlator-option *replicate*.node-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0
>
>Jun 08 06:23:40 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:42 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 09:31:14 Periodic health check status: node 70423cc0bcd044fe5ba8bbbd256a3e49 up=true
>[cmdexec] INFO 2018/06/08 09:31:14 Check Glusterd service status in node dhcp47-76.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 09:31:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 07:25:20 UTC; 2h 5min ago
>  Process: 828 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 830 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5cb2299_6aec_11e8_bc4b_005056a5f18a.slice/docker-7f9223abc9a8641cea68f374f4039235515978ee2dab970ac6cbef182c535426.scope/system.slice/glusterd.service
>           ââ  830 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1197 /usr/sbin/glusterfsd -s 10.70.47.76 --volfile-id heketidbstorage.10.70.47.76.var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.76-var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.pid -S /var/run/gluster/b6976f9ddc3abc90e1dfa91c35a7c611.socket --brick-name /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_54c06cdf52c6fe1e3a3f8a9d39cde33d/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.log --xlator-option *-posix.glusterd-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ13734 /usr/sbin/glusterfsd -s 10.70.47.76 --volfile-id vol_bf31af76ef6c54e7e8f24f4d8711cb22.10.70.47.76.var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_c6935061d1c9dbb05f316e3d87080a38-brick -p /var/run/gluster/vols/vol_bf31af76ef6c54e7e8f24f4d8711cb22/10.70.47.76-var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_c6935061d1c9dbb05f316e3d87080a38-brick.pid -S /var/run/gluster/7aabe8187d4b998d498dd5d4edbd2626.socket --brick-name /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_c6935061d1c9dbb05f316e3d87080a38/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_c6935061d1c9dbb05f316e3d87080a38-brick.log --xlator-option *-posix.glusterd-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94 --brick-port 49153 --xlator-option vol_bf31af76ef6c54e7e8f24f4d8711cb22-server.listen-port=49153
>           ââ18989 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/ec468453b11caed5259f5e1abcc65b44.socket --xlator-option *replicate*.node-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94
>
>Jun 08 07:25:17 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 07:25:20 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Jun 08 08:53:58 dhcp47-76.lab.eng.blr.redhat.com glusterd[830]: [2018-06-08 08:53:58.338441] C [rpc-clnt.c:465:rpc_clnt_fill_request_info] 0-management: cannot lookup the saved frame corresponding to xid (103)
>[heketi] INFO 2018/06/08 09:31:14 Periodic health check status: node d942fe6c0ee5691b7cc263968f97b650 up=true
>[heketi] INFO 2018/06/08 09:31:14 Cleaned 0 nodes from health cache
>[negroni] Started DELETE /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 202 Accepted in 17.798254ms
>[asynchttp] INFO 2018/06/08 09:31:20 asynchttp.go:288: Started job 2d31621f1091a07096dbce4dc581e4f0
>[heketi] INFO 2018/06/08 09:31:20 Started async operation: Delete Volume
>[negroni] Started GET /queue/2d31621f1091a07096dbce4dc581e4f0
>[negroni] Completed 200 OK in 160.609Âµs
>[kubeexec] DEBUG 2018/06/08 09:31:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script snapshot list vol_9c85de66c12db0a72b4d16fe888ff74d --xml
>Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
><cliOutput>
>  <opRet>-1</opRet>
>  <opErrno>30806</opErrno>
>  <opErrstr>Volume (vol_9c85de66c12db0a72b4d16fe888ff74d) does not exist</opErrstr>
></cliOutput>
>[kubeexec] ERROR 2018/06/08 09:31:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume stop vol_9c85de66c12db0a72b4d16fe888ff74d force] on glusterfs-storage-gxp7c: Err[command terminated with exit code 1]: Stdout []: Stderr [volume stop: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>]
>[cmdexec] ERROR 2018/06/08 09:31:21 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:144: Unable to stop volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume stop: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[kubeexec] ERROR 2018/06/08 09:31:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume delete vol_9c85de66c12db0a72b4d16fe888ff74d] on glusterfs-storage-gxp7c: Err[command terminated with exit code 1]: Stdout []: Stderr [volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>]
>[cmdexec] ERROR 2018/06/08 09:31:21 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:153: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[heketi] ERROR 2018/06/08 09:31:21 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:525: Unable to delete volume: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[heketi] ERROR 2018/06/08 09:31:21 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:429: Error executing delete volume: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[asynchttp] INFO 2018/06/08 09:31:21 asynchttp.go:292: Completed job 2d31621f1091a07096dbce4dc581e4f0 in 793.21031ms
>[heketi] ERROR 2018/06/08 09:31:21 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:1185: Delete Volume Failed: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[negroni] Started GET /queue/2d31621f1091a07096dbce4dc581e4f0
>[negroni] Completed 500 Internal Server Error in 283.044Âµs
>[heketi] INFO 2018/06/08 09:33:14 Starting Node Health Status refresh
>[cmdexec] INFO 2018/06/08 09:33:14 Check Glusterd service status in node dhcp46-187.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 09:33:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:33 UTC; 3h 9min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48fed567_6ae4_11e8_bc4b_005056a5f18a.slice/docker-a506d9aaca4b440d184df40d01e90b2040811c8090bf4c19d892081307b28c0c.scope/system.slice/glusterd.service
>           ââ  433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1228 /usr/sbin/glusterfsd -s 10.70.46.187 --volfile-id heketidbstorage.10.70.46.187.var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.187-var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.pid -S /var/run/gluster/c08898c2e3b0190c62df3a9d4a013c92.socket --brick-name /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.log --xlator-option *-posix.glusterd-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ20535 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/8fb729a0bbd51b204491c603c84acaa4.socket --xlator-option *replicate*.node-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034
>
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Unit glusterd.service entered failed state.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: glusterd.service failed.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:33 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 09:33:14 Periodic health check status: node 278bd6b4e16a8e62ef15aaae22e6abc1 up=true
>[cmdexec] INFO 2018/06/08 09:33:14 Check Glusterd service status in node dhcp46-122.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 09:33:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:42 UTC; 3h 9min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4900c968_6ae4_11e8_bc4b_005056a5f18a.slice/docker-438fb5fb16a82a3e701b28fa1f1ef8623dc37ca06374c604e5781a2b2ab2e06d.scope/system.slice/glusterd.service
>           ââ  433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1106 /usr/sbin/glusterfsd -s 10.70.46.122 --volfile-id heketidbstorage.10.70.46.122.var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.122-var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.pid -S /var/run/gluster/9fa3ac3b12826912287adc88c26ac134.socket --brick-name /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_29e5b36d921cca1b0699871d9727c932/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.log --xlator-option *-posix.glusterd-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ21076 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f8994cdb09351d83e70da08261d99c64.socket --xlator-option *replicate*.node-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0
>
>Jun 08 06:23:40 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:42 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 09:33:14 Periodic health check status: node 70423cc0bcd044fe5ba8bbbd256a3e49 up=true
>[cmdexec] INFO 2018/06/08 09:33:14 Check Glusterd service status in node dhcp47-76.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 09:33:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 07:25:20 UTC; 2h 7min ago
>  Process: 828 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 830 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5cb2299_6aec_11e8_bc4b_005056a5f18a.slice/docker-7f9223abc9a8641cea68f374f4039235515978ee2dab970ac6cbef182c535426.scope/system.slice/glusterd.service
>           ââ  830 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1197 /usr/sbin/glusterfsd -s 10.70.47.76 --volfile-id heketidbstorage.10.70.47.76.var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.76-var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.pid -S /var/run/gluster/b6976f9ddc3abc90e1dfa91c35a7c611.socket --brick-name /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_54c06cdf52c6fe1e3a3f8a9d39cde33d/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.log --xlator-option *-posix.glusterd-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ13734 /usr/sbin/glusterfsd -s 10.70.47.76 --volfile-id vol_bf31af76ef6c54e7e8f24f4d8711cb22.10.70.47.76.var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_c6935061d1c9dbb05f316e3d87080a38-brick -p /var/run/gluster/vols/vol_bf31af76ef6c54e7e8f24f4d8711cb22/10.70.47.76-var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_c6935061d1c9dbb05f316e3d87080a38-brick.pid -S /var/run/gluster/7aabe8187d4b998d498dd5d4edbd2626.socket --brick-name /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_c6935061d1c9dbb05f316e3d87080a38/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_c6935061d1c9dbb05f316e3d87080a38-brick.log --xlator-option *-posix.glusterd-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94 --brick-port 49153 --xlator-option vol_bf31af76ef6c54e7e8f24f4d8711cb22-server.listen-port=49153
>           ââ18989 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/ec468453b11caed5259f5e1abcc65b44.socket --xlator-option *replicate*.node-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94
>
>Jun 08 07:25:17 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 07:25:20 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Jun 08 08:53:58 dhcp47-76.lab.eng.blr.redhat.com glusterd[830]: [2018-06-08 08:53:58.338441] C [rpc-clnt.c:465:rpc_clnt_fill_request_info] 0-management: cannot lookup the saved frame corresponding to xid (103)
>[heketi] INFO 2018/06/08 09:33:14 Periodic health check status: node d942fe6c0ee5691b7cc263968f97b650 up=true
>[heketi] INFO 2018/06/08 09:33:14 Cleaned 0 nodes from health cache
>[negroni] Started DELETE /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 202 Accepted in 25.540196ms
>[asynchttp] INFO 2018/06/08 09:33:20 asynchttp.go:288: Started job c2d4c096e925be6f447c2ae998426ef2
>[heketi] INFO 2018/06/08 09:33:20 Started async operation: Delete Volume
>[negroni] Started GET /queue/c2d4c096e925be6f447c2ae998426ef2
>[negroni] Completed 200 OK in 178.048Âµs
>[kubeexec] DEBUG 2018/06/08 09:33:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script snapshot list vol_5db55c483f6984d9917cfe4b3c8b3cbc --xml
>Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
><cliOutput>
>  <opRet>-1</opRet>
>  <opErrno>30806</opErrno>
>  <opErrstr>Volume (vol_5db55c483f6984d9917cfe4b3c8b3cbc) does not exist</opErrstr>
></cliOutput>
>[kubeexec] ERROR 2018/06/08 09:33:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume stop vol_5db55c483f6984d9917cfe4b3c8b3cbc force] on glusterfs-storage-pg4xc: Err[command terminated with exit code 1]: Stdout []: Stderr [volume stop: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>]
>[cmdexec] ERROR 2018/06/08 09:33:21 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:144: Unable to stop volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume stop: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[kubeexec] ERROR 2018/06/08 09:33:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume delete vol_5db55c483f6984d9917cfe4b3c8b3cbc] on glusterfs-storage-pg4xc: Err[command terminated with exit code 1]: Stdout []: Stderr [volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>]
>[cmdexec] ERROR 2018/06/08 09:33:21 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:153: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[heketi] ERROR 2018/06/08 09:33:21 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:525: Unable to delete volume: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[heketi] ERROR 2018/06/08 09:33:21 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:429: Error executing delete volume: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[asynchttp] INFO 2018/06/08 09:33:21 asynchttp.go:292: Completed job c2d4c096e925be6f447c2ae998426ef2 in 740.28419ms
>[heketi] ERROR 2018/06/08 09:33:21 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:1185: Delete Volume Failed: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[negroni] Started GET /queue/c2d4c096e925be6f447c2ae998426ef2
>[negroni] Completed 500 Internal Server Error in 142.768Âµs
>[negroni] Started DELETE /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 202 Accepted in 19.328309ms
>[asynchttp] INFO 2018/06/08 09:33:35 asynchttp.go:288: Started job a499293e1304c9f67982b8be149d2217
>[heketi] INFO 2018/06/08 09:33:35 Started async operation: Delete Volume
>[negroni] Started GET /queue/a499293e1304c9f67982b8be149d2217
>[negroni] Completed 200 OK in 120.546Âµs
>[kubeexec] DEBUG 2018/06/08 09:33:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script snapshot list vol_9c85de66c12db0a72b4d16fe888ff74d --xml
>Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
><cliOutput>
>  <opRet>-1</opRet>
>  <opErrno>30806</opErrno>
>  <opErrstr>Volume (vol_9c85de66c12db0a72b4d16fe888ff74d) does not exist</opErrstr>
></cliOutput>
>[kubeexec] ERROR 2018/06/08 09:33:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume stop vol_9c85de66c12db0a72b4d16fe888ff74d force] on glusterfs-storage-gxp7c: Err[command terminated with exit code 1]: Stdout []: Stderr [volume stop: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>]
>[cmdexec] ERROR 2018/06/08 09:33:36 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:144: Unable to stop volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume stop: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[kubeexec] ERROR 2018/06/08 09:33:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume delete vol_9c85de66c12db0a72b4d16fe888ff74d] on glusterfs-storage-gxp7c: Err[command terminated with exit code 1]: Stdout []: Stderr [volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>]
>[cmdexec] ERROR 2018/06/08 09:33:36 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:153: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[heketi] ERROR 2018/06/08 09:33:36 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:525: Unable to delete volume: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[heketi] ERROR 2018/06/08 09:33:36 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:429: Error executing delete volume: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[heketi] ERROR 2018/06/08 09:33:36 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:1185: Delete Volume Failed: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[asynchttp] INFO 2018/06/08 09:33:36 asynchttp.go:292: Completed job a499293e1304c9f67982b8be149d2217 in 766.3984ms
>[negroni] Started GET /queue/a499293e1304c9f67982b8be149d2217
>[negroni] Completed 500 Internal Server Error in 366.906Âµs
>[heketi] INFO 2018/06/08 09:35:14 Starting Node Health Status refresh
>[cmdexec] INFO 2018/06/08 09:35:14 Check Glusterd service status in node dhcp46-187.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 09:35:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:33 UTC; 3h 11min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48fed567_6ae4_11e8_bc4b_005056a5f18a.slice/docker-a506d9aaca4b440d184df40d01e90b2040811c8090bf4c19d892081307b28c0c.scope/system.slice/glusterd.service
>           ââ  433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1228 /usr/sbin/glusterfsd -s 10.70.46.187 --volfile-id heketidbstorage.10.70.46.187.var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.187-var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.pid -S /var/run/gluster/c08898c2e3b0190c62df3a9d4a013c92.socket --brick-name /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.log --xlator-option *-posix.glusterd-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ20535 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/8fb729a0bbd51b204491c603c84acaa4.socket --xlator-option *replicate*.node-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034
>
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Unit glusterd.service entered failed state.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: glusterd.service failed.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:33 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 09:35:14 Periodic health check status: node 278bd6b4e16a8e62ef15aaae22e6abc1 up=true
>[cmdexec] INFO 2018/06/08 09:35:14 Check Glusterd service status in node dhcp46-122.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 09:35:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:42 UTC; 3h 11min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4900c968_6ae4_11e8_bc4b_005056a5f18a.slice/docker-438fb5fb16a82a3e701b28fa1f1ef8623dc37ca06374c604e5781a2b2ab2e06d.scope/system.slice/glusterd.service
>           ââ  433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1106 /usr/sbin/glusterfsd -s 10.70.46.122 --volfile-id heketidbstorage.10.70.46.122.var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.122-var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.pid -S /var/run/gluster/9fa3ac3b12826912287adc88c26ac134.socket --brick-name /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_29e5b36d921cca1b0699871d9727c932/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.log --xlator-option *-posix.glusterd-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ21076 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f8994cdb09351d83e70da08261d99c64.socket --xlator-option *replicate*.node-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0
>
>Jun 08 06:23:40 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:42 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 09:35:14 Periodic health check status: node 70423cc0bcd044fe5ba8bbbd256a3e49 up=true
>[cmdexec] INFO 2018/06/08 09:35:14 Check Glusterd service status in node dhcp47-76.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 09:35:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 07:25:20 UTC; 2h 9min ago
>  Process: 828 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 830 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5cb2299_6aec_11e8_bc4b_005056a5f18a.slice/docker-7f9223abc9a8641cea68f374f4039235515978ee2dab970ac6cbef182c535426.scope/system.slice/glusterd.service
>           ââ  830 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1197 /usr/sbin/glusterfsd -s 10.70.47.76 --volfile-id heketidbstorage.10.70.47.76.var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.76-var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.pid -S /var/run/gluster/b6976f9ddc3abc90e1dfa91c35a7c611.socket --brick-name /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_54c06cdf52c6fe1e3a3f8a9d39cde33d/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.log --xlator-option *-posix.glusterd-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ13734 /usr/sbin/glusterfsd -s 10.70.47.76 --volfile-id vol_bf31af76ef6c54e7e8f24f4d8711cb22.10.70.47.76.var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_c6935061d1c9dbb05f316e3d87080a38-brick -p /var/run/gluster/vols/vol_bf31af76ef6c54e7e8f24f4d8711cb22/10.70.47.76-var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_c6935061d1c9dbb05f316e3d87080a38-brick.pid -S /var/run/gluster/7aabe8187d4b998d498dd5d4edbd2626.socket --brick-name /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_c6935061d1c9dbb05f316e3d87080a38/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_c6935061d1c9dbb05f316e3d87080a38-brick.log --xlator-option *-posix.glusterd-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94 --brick-port 49153 --xlator-option vol_bf31af76ef6c54e7e8f24f4d8711cb22-server.listen-port=49153
>           ââ18989 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/ec468453b11caed5259f5e1abcc65b44.socket --xlator-option *replicate*.node-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94
>
>Jun 08 07:25:17 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 07:25:20 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Jun 08 08:53:58 dhcp47-76.lab.eng.blr.redhat.com glusterd[830]: [2018-06-08 08:53:58.338441] C [rpc-clnt.c:465:rpc_clnt_fill_request_info] 0-management: cannot lookup the saved frame corresponding to xid (103)
>[heketi] INFO 2018/06/08 09:35:14 Periodic health check status: node d942fe6c0ee5691b7cc263968f97b650 up=true
>[heketi] INFO 2018/06/08 09:35:14 Cleaned 0 nodes from health cache
>[negroni] Started DELETE /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 202 Accepted in 29.803998ms
>[asynchttp] INFO 2018/06/08 09:35:35 asynchttp.go:288: Started job ac3d999d5150760dc7ab46387dd6e55f
>[heketi] INFO 2018/06/08 09:35:35 Started async operation: Delete Volume
>[negroni] Started GET /queue/ac3d999d5150760dc7ab46387dd6e55f
>[negroni] Completed 200 OK in 201.549Âµs
>[kubeexec] DEBUG 2018/06/08 09:35:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script snapshot list vol_5db55c483f6984d9917cfe4b3c8b3cbc --xml
>Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
><cliOutput>
>  <opRet>-1</opRet>
>  <opErrno>30806</opErrno>
>  <opErrstr>Volume (vol_5db55c483f6984d9917cfe4b3c8b3cbc) does not exist</opErrstr>
></cliOutput>
>[kubeexec] ERROR 2018/06/08 09:35:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume stop vol_5db55c483f6984d9917cfe4b3c8b3cbc force] on glusterfs-storage-pg4xc: Err[command terminated with exit code 1]: Stdout []: Stderr [volume stop: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>]
>[cmdexec] ERROR 2018/06/08 09:35:36 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:144: Unable to stop volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume stop: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[kubeexec] ERROR 2018/06/08 09:35:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume delete vol_5db55c483f6984d9917cfe4b3c8b3cbc] on glusterfs-storage-pg4xc: Err[command terminated with exit code 1]: Stdout []: Stderr [volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>]
>[cmdexec] ERROR 2018/06/08 09:35:36 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:153: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[heketi] ERROR 2018/06/08 09:35:36 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:525: Unable to delete volume: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[heketi] ERROR 2018/06/08 09:35:36 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:429: Error executing delete volume: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[asynchttp] INFO 2018/06/08 09:35:36 asynchttp.go:292: Completed job ac3d999d5150760dc7ab46387dd6e55f in 751.999697ms
>[heketi] ERROR 2018/06/08 09:35:36 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:1185: Delete Volume Failed: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[negroni] Started GET /queue/ac3d999d5150760dc7ab46387dd6e55f
>[negroni] Completed 500 Internal Server Error in 184.613Âµs
>[negroni] Started DELETE /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 202 Accepted in 30.364725ms
>[asynchttp] INFO 2018/06/08 09:35:50 asynchttp.go:288: Started job b618307c047c4b54578a12d534399dae
>[heketi] INFO 2018/06/08 09:35:50 Started async operation: Delete Volume
>[negroni] Started GET /queue/b618307c047c4b54578a12d534399dae
>[negroni] Completed 200 OK in 173.858Âµs
>[kubeexec] DEBUG 2018/06/08 09:35:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script snapshot list vol_9c85de66c12db0a72b4d16fe888ff74d --xml
>Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
><cliOutput>
>  <opRet>-1</opRet>
>  <opErrno>30806</opErrno>
>  <opErrstr>Volume (vol_9c85de66c12db0a72b4d16fe888ff74d) does not exist</opErrstr>
></cliOutput>
>[kubeexec] ERROR 2018/06/08 09:35:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume stop vol_9c85de66c12db0a72b4d16fe888ff74d force] on glusterfs-storage-gxp7c: Err[command terminated with exit code 1]: Stdout []: Stderr [volume stop: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>]
>[cmdexec] ERROR 2018/06/08 09:35:51 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:144: Unable to stop volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume stop: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[kubeexec] ERROR 2018/06/08 09:35:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume delete vol_9c85de66c12db0a72b4d16fe888ff74d] on glusterfs-storage-gxp7c: Err[command terminated with exit code 1]: Stdout []: Stderr [volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>]
>[cmdexec] ERROR 2018/06/08 09:35:51 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:153: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[heketi] ERROR 2018/06/08 09:35:51 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:525: Unable to delete volume: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[heketi] ERROR 2018/06/08 09:35:51 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:429: Error executing delete volume: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[heketi] ERROR 2018/06/08 09:35:51 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:1185: Delete Volume Failed: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[asynchttp] INFO 2018/06/08 09:35:51 asynchttp.go:292: Completed job b618307c047c4b54578a12d534399dae in 773.380988ms
>[negroni] Started GET /queue/b618307c047c4b54578a12d534399dae
>[negroni] Completed 500 Internal Server Error in 192.758Âµs
>[heketi] INFO 2018/06/08 09:37:14 Starting Node Health Status refresh
>[cmdexec] INFO 2018/06/08 09:37:14 Check Glusterd service status in node dhcp46-187.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 09:37:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:33 UTC; 3h 13min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48fed567_6ae4_11e8_bc4b_005056a5f18a.slice/docker-a506d9aaca4b440d184df40d01e90b2040811c8090bf4c19d892081307b28c0c.scope/system.slice/glusterd.service
>           ââ  433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1228 /usr/sbin/glusterfsd -s 10.70.46.187 --volfile-id heketidbstorage.10.70.46.187.var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.187-var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.pid -S /var/run/gluster/c08898c2e3b0190c62df3a9d4a013c92.socket --brick-name /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.log --xlator-option *-posix.glusterd-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ20535 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/8fb729a0bbd51b204491c603c84acaa4.socket --xlator-option *replicate*.node-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034
>
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Unit glusterd.service entered failed state.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: glusterd.service failed.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:33 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 09:37:14 Periodic health check status: node 278bd6b4e16a8e62ef15aaae22e6abc1 up=true
>[cmdexec] INFO 2018/06/08 09:37:14 Check Glusterd service status in node dhcp46-122.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 09:37:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:42 UTC; 3h 13min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4900c968_6ae4_11e8_bc4b_005056a5f18a.slice/docker-438fb5fb16a82a3e701b28fa1f1ef8623dc37ca06374c604e5781a2b2ab2e06d.scope/system.slice/glusterd.service
>           ââ  433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1106 /usr/sbin/glusterfsd -s 10.70.46.122 --volfile-id heketidbstorage.10.70.46.122.var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.122-var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.pid -S /var/run/gluster/9fa3ac3b12826912287adc88c26ac134.socket --brick-name /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_29e5b36d921cca1b0699871d9727c932/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.log --xlator-option *-posix.glusterd-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ21076 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f8994cdb09351d83e70da08261d99c64.socket --xlator-option *replicate*.node-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0
>
>Jun 08 06:23:40 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:42 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 09:37:14 Periodic health check status: node 70423cc0bcd044fe5ba8bbbd256a3e49 up=true
>[cmdexec] INFO 2018/06/08 09:37:14 Check Glusterd service status in node dhcp47-76.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 09:37:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 07:25:20 UTC; 2h 11min ago
>  Process: 828 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 830 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5cb2299_6aec_11e8_bc4b_005056a5f18a.slice/docker-7f9223abc9a8641cea68f374f4039235515978ee2dab970ac6cbef182c535426.scope/system.slice/glusterd.service
>           ââ  830 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1197 /usr/sbin/glusterfsd -s 10.70.47.76 --volfile-id heketidbstorage.10.70.47.76.var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.76-var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.pid -S /var/run/gluster/b6976f9ddc3abc90e1dfa91c35a7c611.socket --brick-name /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_54c06cdf52c6fe1e3a3f8a9d39cde33d/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.log --xlator-option *-posix.glusterd-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ13734 /usr/sbin/glusterfsd -s 10.70.47.76 --volfile-id vol_bf31af76ef6c54e7e8f24f4d8711cb22.10.70.47.76.var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_c6935061d1c9dbb05f316e3d87080a38-brick -p /var/run/gluster/vols/vol_bf31af76ef6c54e7e8f24f4d8711cb22/10.70.47.76-var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_c6935061d1c9dbb05f316e3d87080a38-brick.pid -S /var/run/gluster/7aabe8187d4b998d498dd5d4edbd2626.socket --brick-name /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_c6935061d1c9dbb05f316e3d87080a38/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_c6935061d1c9dbb05f316e3d87080a38-brick.log --xlator-option *-posix.glusterd-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94 --brick-port 49153 --xlator-option vol_bf31af76ef6c54e7e8f24f4d8711cb22-server.listen-port=49153
>           ââ18989 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/ec468453b11caed5259f5e1abcc65b44.socket --xlator-option *replicate*.node-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94
>
>Jun 08 07:25:17 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 07:25:20 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Jun 08 08:53:58 dhcp47-76.lab.eng.blr.redhat.com glusterd[830]: [2018-06-08 08:53:58.338441] C [rpc-clnt.c:465:rpc_clnt_fill_request_info] 0-management: cannot lookup the saved frame corresponding to xid (103)
>[heketi] INFO 2018/06/08 09:37:14 Periodic health check status: node d942fe6c0ee5691b7cc263968f97b650 up=true
>[heketi] INFO 2018/06/08 09:37:14 Cleaned 0 nodes from health cache
>[negroni] Started DELETE /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 202 Accepted in 33.841851ms
>[asynchttp] INFO 2018/06/08 09:37:50 asynchttp.go:288: Started job ca99e89505f5d0eb0db763e7d6005e54
>[heketi] INFO 2018/06/08 09:37:50 Started async operation: Delete Volume
>[negroni] Started GET /queue/ca99e89505f5d0eb0db763e7d6005e54
>[negroni] Completed 200 OK in 181.622Âµs
>[kubeexec] DEBUG 2018/06/08 09:37:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script snapshot list vol_5db55c483f6984d9917cfe4b3c8b3cbc --xml
>Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
><cliOutput>
>  <opRet>-1</opRet>
>  <opErrno>30806</opErrno>
>  <opErrstr>Volume (vol_5db55c483f6984d9917cfe4b3c8b3cbc) does not exist</opErrstr>
></cliOutput>
>[kubeexec] ERROR 2018/06/08 09:37:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume stop vol_5db55c483f6984d9917cfe4b3c8b3cbc force] on glusterfs-storage-pg4xc: Err[command terminated with exit code 1]: Stdout []: Stderr [volume stop: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>]
>[cmdexec] ERROR 2018/06/08 09:37:51 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:144: Unable to stop volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume stop: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[kubeexec] ERROR 2018/06/08 09:37:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume delete vol_5db55c483f6984d9917cfe4b3c8b3cbc] on glusterfs-storage-pg4xc: Err[command terminated with exit code 1]: Stdout []: Stderr [volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>]
>[cmdexec] ERROR 2018/06/08 09:37:51 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:153: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[heketi] ERROR 2018/06/08 09:37:51 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:525: Unable to delete volume: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[heketi] ERROR 2018/06/08 09:37:51 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:429: Error executing delete volume: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[heketi] ERROR 2018/06/08 09:37:51 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:1185: Delete Volume Failed: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[asynchttp] INFO 2018/06/08 09:37:51 asynchttp.go:292: Completed job ca99e89505f5d0eb0db763e7d6005e54 in 805.641713ms
>[negroni] Started GET /queue/ca99e89505f5d0eb0db763e7d6005e54
>[negroni] Completed 500 Internal Server Error in 137.435Âµs
>[negroni] Started DELETE /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 202 Accepted in 24.741111ms
>[asynchttp] INFO 2018/06/08 09:38:05 asynchttp.go:288: Started job d9257a81c6c1968a88ffa07c728881e3
>[heketi] INFO 2018/06/08 09:38:05 Started async operation: Delete Volume
>[negroni] Started GET /queue/d9257a81c6c1968a88ffa07c728881e3
>[negroni] Completed 200 OK in 190.371Âµs
>[kubeexec] DEBUG 2018/06/08 09:38:06 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script snapshot list vol_9c85de66c12db0a72b4d16fe888ff74d --xml
>Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
><cliOutput>
>  <opRet>-1</opRet>
>  <opErrno>30806</opErrno>
>  <opErrstr>Volume (vol_9c85de66c12db0a72b4d16fe888ff74d) does not exist</opErrstr>
></cliOutput>
>[kubeexec] ERROR 2018/06/08 09:38:06 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume stop vol_9c85de66c12db0a72b4d16fe888ff74d force] on glusterfs-storage-gxp7c: Err[command terminated with exit code 1]: Stdout []: Stderr [volume stop: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>]
>[cmdexec] ERROR 2018/06/08 09:38:06 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:144: Unable to stop volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume stop: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[kubeexec] ERROR 2018/06/08 09:38:06 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume delete vol_9c85de66c12db0a72b4d16fe888ff74d] on glusterfs-storage-gxp7c: Err[command terminated with exit code 1]: Stdout []: Stderr [volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>]
>[cmdexec] ERROR 2018/06/08 09:38:06 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:153: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[heketi] ERROR 2018/06/08 09:38:06 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:525: Unable to delete volume: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[heketi] ERROR 2018/06/08 09:38:06 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:429: Error executing delete volume: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[asynchttp] INFO 2018/06/08 09:38:06 asynchttp.go:292: Completed job d9257a81c6c1968a88ffa07c728881e3 in 767.315424ms
>[heketi] ERROR 2018/06/08 09:38:06 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:1185: Delete Volume Failed: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[negroni] Started GET /queue/d9257a81c6c1968a88ffa07c728881e3
>[negroni] Completed 500 Internal Server Error in 240.335Âµs
>[heketi] INFO 2018/06/08 09:39:14 Starting Node Health Status refresh
>[cmdexec] INFO 2018/06/08 09:39:14 Check Glusterd service status in node dhcp46-187.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 09:39:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:33 UTC; 3h 15min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48fed567_6ae4_11e8_bc4b_005056a5f18a.slice/docker-a506d9aaca4b440d184df40d01e90b2040811c8090bf4c19d892081307b28c0c.scope/system.slice/glusterd.service
>           ââ  433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1228 /usr/sbin/glusterfsd -s 10.70.46.187 --volfile-id heketidbstorage.10.70.46.187.var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.187-var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.pid -S /var/run/gluster/c08898c2e3b0190c62df3a9d4a013c92.socket --brick-name /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.log --xlator-option *-posix.glusterd-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ20535 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/8fb729a0bbd51b204491c603c84acaa4.socket --xlator-option *replicate*.node-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034
>
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Unit glusterd.service entered failed state.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: glusterd.service failed.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:33 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 09:39:14 Periodic health check status: node 278bd6b4e16a8e62ef15aaae22e6abc1 up=true
>[cmdexec] INFO 2018/06/08 09:39:14 Check Glusterd service status in node dhcp46-122.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 09:39:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:42 UTC; 3h 15min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4900c968_6ae4_11e8_bc4b_005056a5f18a.slice/docker-438fb5fb16a82a3e701b28fa1f1ef8623dc37ca06374c604e5781a2b2ab2e06d.scope/system.slice/glusterd.service
>           ââ  433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1106 /usr/sbin/glusterfsd -s 10.70.46.122 --volfile-id heketidbstorage.10.70.46.122.var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.122-var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.pid -S /var/run/gluster/9fa3ac3b12826912287adc88c26ac134.socket --brick-name /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_29e5b36d921cca1b0699871d9727c932/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.log --xlator-option *-posix.glusterd-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ21076 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f8994cdb09351d83e70da08261d99c64.socket --xlator-option *replicate*.node-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0
>
>Jun 08 06:23:40 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:42 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 09:39:14 Periodic health check status: node 70423cc0bcd044fe5ba8bbbd256a3e49 up=true
>[cmdexec] INFO 2018/06/08 09:39:14 Check Glusterd service status in node dhcp47-76.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 09:39:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 07:25:20 UTC; 2h 13min ago
>  Process: 828 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 830 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5cb2299_6aec_11e8_bc4b_005056a5f18a.slice/docker-7f9223abc9a8641cea68f374f4039235515978ee2dab970ac6cbef182c535426.scope/system.slice/glusterd.service
>           ââ  830 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1197 /usr/sbin/glusterfsd -s 10.70.47.76 --volfile-id heketidbstorage.10.70.47.76.var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.76-var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.pid -S /var/run/gluster/b6976f9ddc3abc90e1dfa91c35a7c611.socket --brick-name /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_54c06cdf52c6fe1e3a3f8a9d39cde33d/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.log --xlator-option *-posix.glusterd-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ13734 /usr/sbin/glusterfsd -s 10.70.47.76 --volfile-id vol_bf31af76ef6c54e7e8f24f4d8711cb22.10.70.47.76.var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_c6935061d1c9dbb05f316e3d87080a38-brick -p /var/run/gluster/vols/vol_bf31af76ef6c54e7e8f24f4d8711cb22/10.70.47.76-var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_c6935061d1c9dbb05f316e3d87080a38-brick.pid -S /var/run/gluster/7aabe8187d4b998d498dd5d4edbd2626.socket --brick-name /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_c6935061d1c9dbb05f316e3d87080a38/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_c6935061d1c9dbb05f316e3d87080a38-brick.log --xlator-option *-posix.glusterd-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94 --brick-port 49153 --xlator-option vol_bf31af76ef6c54e7e8f24f4d8711cb22-server.listen-port=49153
>           ââ18989 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/ec468453b11caed5259f5e1abcc65b44.socket --xlator-option *replicate*.node-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94
>
>Jun 08 07:25:17 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 07:25:20 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Jun 08 08:53:58 dhcp47-76.lab.eng.blr.redhat.com glusterd[830]: [2018-06-08 08:53:58.338441] C [rpc-clnt.c:465:rpc_clnt_fill_request_info] 0-management: cannot lookup the saved frame corresponding to xid (103)
>[heketi] INFO 2018/06/08 09:39:14 Periodic health check status: node d942fe6c0ee5691b7cc263968f97b650 up=true
>[heketi] INFO 2018/06/08 09:39:14 Cleaned 0 nodes from health cache
>[negroni] Started DELETE /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 202 Accepted in 19.771509ms
>[asynchttp] INFO 2018/06/08 09:40:05 asynchttp.go:288: Started job d51bb402b86fdf52fd5cd76853c9c9cc
>[heketi] INFO 2018/06/08 09:40:05 Started async operation: Delete Volume
>[negroni] Started GET /queue/d51bb402b86fdf52fd5cd76853c9c9cc
>[negroni] Completed 200 OK in 145.967Âµs
>[kubeexec] DEBUG 2018/06/08 09:40:06 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script snapshot list vol_5db55c483f6984d9917cfe4b3c8b3cbc --xml
>Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
><cliOutput>
>  <opRet>-1</opRet>
>  <opErrno>30806</opErrno>
>  <opErrstr>Volume (vol_5db55c483f6984d9917cfe4b3c8b3cbc) does not exist</opErrstr>
></cliOutput>
>[kubeexec] ERROR 2018/06/08 09:40:06 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume stop vol_5db55c483f6984d9917cfe4b3c8b3cbc force] on glusterfs-storage-pg4xc: Err[command terminated with exit code 1]: Stdout []: Stderr [volume stop: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>]
>[cmdexec] ERROR 2018/06/08 09:40:06 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:144: Unable to stop volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume stop: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[kubeexec] ERROR 2018/06/08 09:40:06 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume delete vol_5db55c483f6984d9917cfe4b3c8b3cbc] on glusterfs-storage-pg4xc: Err[command terminated with exit code 1]: Stdout []: Stderr [volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>]
>[cmdexec] ERROR 2018/06/08 09:40:06 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:153: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[heketi] ERROR 2018/06/08 09:40:06 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:525: Unable to delete volume: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[heketi] ERROR 2018/06/08 09:40:06 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:429: Error executing delete volume: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[asynchttp] INFO 2018/06/08 09:40:06 asynchttp.go:292: Completed job d51bb402b86fdf52fd5cd76853c9c9cc in 814.931023ms
>[heketi] ERROR 2018/06/08 09:40:06 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:1185: Delete Volume Failed: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[negroni] Started GET /queue/d51bb402b86fdf52fd5cd76853c9c9cc
>[negroni] Completed 500 Internal Server Error in 279.118Âµs
>[negroni] Started DELETE /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 202 Accepted in 28.397208ms
>[asynchttp] INFO 2018/06/08 09:40:20 asynchttp.go:288: Started job d4f67b69f3e6b075c72d0efac5f1bbe4
>[heketi] INFO 2018/06/08 09:40:20 Started async operation: Delete Volume
>[negroni] Started GET /queue/d4f67b69f3e6b075c72d0efac5f1bbe4
>[negroni] Completed 200 OK in 175.312Âµs
>[kubeexec] DEBUG 2018/06/08 09:40:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script snapshot list vol_9c85de66c12db0a72b4d16fe888ff74d --xml
>Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
><cliOutput>
>  <opRet>-1</opRet>
>  <opErrno>30806</opErrno>
>  <opErrstr>Volume (vol_9c85de66c12db0a72b4d16fe888ff74d) does not exist</opErrstr>
></cliOutput>
>[kubeexec] ERROR 2018/06/08 09:40:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume stop vol_9c85de66c12db0a72b4d16fe888ff74d force] on glusterfs-storage-gxp7c: Err[command terminated with exit code 1]: Stdout []: Stderr [volume stop: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>]
>[cmdexec] ERROR 2018/06/08 09:40:21 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:144: Unable to stop volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume stop: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[kubeexec] ERROR 2018/06/08 09:40:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume delete vol_9c85de66c12db0a72b4d16fe888ff74d] on glusterfs-storage-gxp7c: Err[command terminated with exit code 1]: Stdout []: Stderr [volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>]
>[cmdexec] ERROR 2018/06/08 09:40:21 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:153: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[heketi] ERROR 2018/06/08 09:40:21 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:525: Unable to delete volume: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[heketi] ERROR 2018/06/08 09:40:21 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:429: Error executing delete volume: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[heketi] ERROR 2018/06/08 09:40:21 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:1185: Delete Volume Failed: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[asynchttp] INFO 2018/06/08 09:40:21 asynchttp.go:292: Completed job d4f67b69f3e6b075c72d0efac5f1bbe4 in 769.429683ms
>[negroni] Started GET /queue/d4f67b69f3e6b075c72d0efac5f1bbe4
>[negroni] Completed 500 Internal Server Error in 127.35Âµs
>[heketi] INFO 2018/06/08 09:41:14 Starting Node Health Status refresh
>[cmdexec] INFO 2018/06/08 09:41:14 Check Glusterd service status in node dhcp46-187.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 09:41:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:33 UTC; 3h 17min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48fed567_6ae4_11e8_bc4b_005056a5f18a.slice/docker-a506d9aaca4b440d184df40d01e90b2040811c8090bf4c19d892081307b28c0c.scope/system.slice/glusterd.service
>           ââ  433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1228 /usr/sbin/glusterfsd -s 10.70.46.187 --volfile-id heketidbstorage.10.70.46.187.var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.187-var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.pid -S /var/run/gluster/c08898c2e3b0190c62df3a9d4a013c92.socket --brick-name /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.log --xlator-option *-posix.glusterd-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ20535 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/8fb729a0bbd51b204491c603c84acaa4.socket --xlator-option *replicate*.node-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034
>
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Unit glusterd.service entered failed state.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: glusterd.service failed.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:33 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 09:41:14 Periodic health check status: node 278bd6b4e16a8e62ef15aaae22e6abc1 up=true
>[cmdexec] INFO 2018/06/08 09:41:14 Check Glusterd service status in node dhcp46-122.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 09:41:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:42 UTC; 3h 17min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4900c968_6ae4_11e8_bc4b_005056a5f18a.slice/docker-438fb5fb16a82a3e701b28fa1f1ef8623dc37ca06374c604e5781a2b2ab2e06d.scope/system.slice/glusterd.service
>           ââ  433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1106 /usr/sbin/glusterfsd -s 10.70.46.122 --volfile-id heketidbstorage.10.70.46.122.var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.122-var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.pid -S /var/run/gluster/9fa3ac3b12826912287adc88c26ac134.socket --brick-name /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_29e5b36d921cca1b0699871d9727c932/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.log --xlator-option *-posix.glusterd-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ21076 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f8994cdb09351d83e70da08261d99c64.socket --xlator-option *replicate*.node-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0
>
>Jun 08 06:23:40 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:42 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 09:41:14 Periodic health check status: node 70423cc0bcd044fe5ba8bbbd256a3e49 up=true
>[cmdexec] INFO 2018/06/08 09:41:14 Check Glusterd service status in node dhcp47-76.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 09:41:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 07:25:20 UTC; 2h 15min ago
>  Process: 828 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 830 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5cb2299_6aec_11e8_bc4b_005056a5f18a.slice/docker-7f9223abc9a8641cea68f374f4039235515978ee2dab970ac6cbef182c535426.scope/system.slice/glusterd.service
>           ââ  830 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1197 /usr/sbin/glusterfsd -s 10.70.47.76 --volfile-id heketidbstorage.10.70.47.76.var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.76-var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.pid -S /var/run/gluster/b6976f9ddc3abc90e1dfa91c35a7c611.socket --brick-name /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_54c06cdf52c6fe1e3a3f8a9d39cde33d/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.log --xlator-option *-posix.glusterd-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ13734 /usr/sbin/glusterfsd -s 10.70.47.76 --volfile-id vol_bf31af76ef6c54e7e8f24f4d8711cb22.10.70.47.76.var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_c6935061d1c9dbb05f316e3d87080a38-brick -p /var/run/gluster/vols/vol_bf31af76ef6c54e7e8f24f4d8711cb22/10.70.47.76-var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_c6935061d1c9dbb05f316e3d87080a38-brick.pid -S /var/run/gluster/7aabe8187d4b998d498dd5d4edbd2626.socket --brick-name /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_c6935061d1c9dbb05f316e3d87080a38/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_c6935061d1c9dbb05f316e3d87080a38-brick.log --xlator-option *-posix.glusterd-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94 --brick-port 49153 --xlator-option vol_bf31af76ef6c54e7e8f24f4d8711cb22-server.listen-port=49153
>           ââ18989 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/ec468453b11caed5259f5e1abcc65b44.socket --xlator-option *replicate*.node-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94
>
>Jun 08 07:25:17 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 07:25:20 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Jun 08 08:53:58 dhcp47-76.lab.eng.blr.redhat.com glusterd[830]: [2018-06-08 08:53:58.338441] C [rpc-clnt.c:465:rpc_clnt_fill_request_info] 0-management: cannot lookup the saved frame corresponding to xid (103)
>[heketi] INFO 2018/06/08 09:41:14 Periodic health check status: node d942fe6c0ee5691b7cc263968f97b650 up=true
>[heketi] INFO 2018/06/08 09:41:14 Cleaned 0 nodes from health cache
>[negroni] Started DELETE /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 202 Accepted in 27.846178ms
>[asynchttp] INFO 2018/06/08 09:42:20 asynchttp.go:288: Started job aefbe0a3936e46d0016846316d349426
>[heketi] INFO 2018/06/08 09:42:20 Started async operation: Delete Volume
>[negroni] Started GET /queue/aefbe0a3936e46d0016846316d349426
>[negroni] Completed 200 OK in 170.571Âµs
>[kubeexec] DEBUG 2018/06/08 09:42:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script snapshot list vol_5db55c483f6984d9917cfe4b3c8b3cbc --xml
>Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
><cliOutput>
>  <opRet>-1</opRet>
>  <opErrno>30806</opErrno>
>  <opErrstr>Volume (vol_5db55c483f6984d9917cfe4b3c8b3cbc) does not exist</opErrstr>
></cliOutput>
>[kubeexec] ERROR 2018/06/08 09:42:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume stop vol_5db55c483f6984d9917cfe4b3c8b3cbc force] on glusterfs-storage-pg4xc: Err[command terminated with exit code 1]: Stdout []: Stderr [volume stop: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>]
>[cmdexec] ERROR 2018/06/08 09:42:21 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:144: Unable to stop volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume stop: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[kubeexec] ERROR 2018/06/08 09:42:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume delete vol_5db55c483f6984d9917cfe4b3c8b3cbc] on glusterfs-storage-pg4xc: Err[command terminated with exit code 1]: Stdout []: Stderr [volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>]
>[cmdexec] ERROR 2018/06/08 09:42:21 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:153: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[heketi] ERROR 2018/06/08 09:42:21 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:525: Unable to delete volume: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[heketi] ERROR 2018/06/08 09:42:21 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:429: Error executing delete volume: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[heketi] ERROR 2018/06/08 09:42:21 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:1185: Delete Volume Failed: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[asynchttp] INFO 2018/06/08 09:42:21 asynchttp.go:292: Completed job aefbe0a3936e46d0016846316d349426 in 819.987698ms
>[negroni] Started GET /queue/aefbe0a3936e46d0016846316d349426
>[negroni] Completed 500 Internal Server Error in 149.111Âµs
>[negroni] Started DELETE /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 202 Accepted in 20.758907ms
>[asynchttp] INFO 2018/06/08 09:42:35 asynchttp.go:288: Started job c90347d62327f9658943dbc7cb811e91
>[heketi] INFO 2018/06/08 09:42:35 Started async operation: Delete Volume
>[negroni] Started GET /queue/c90347d62327f9658943dbc7cb811e91
>[negroni] Completed 200 OK in 179.143Âµs
>[kubeexec] DEBUG 2018/06/08 09:42:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script snapshot list vol_9c85de66c12db0a72b4d16fe888ff74d --xml
>Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
><cliOutput>
>  <opRet>-1</opRet>
>  <opErrno>30806</opErrno>
>  <opErrstr>Volume (vol_9c85de66c12db0a72b4d16fe888ff74d) does not exist</opErrstr>
></cliOutput>
>[kubeexec] ERROR 2018/06/08 09:42:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume stop vol_9c85de66c12db0a72b4d16fe888ff74d force] on glusterfs-storage-gxp7c: Err[command terminated with exit code 1]: Stdout []: Stderr [volume stop: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>]
>[cmdexec] ERROR 2018/06/08 09:42:36 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:144: Unable to stop volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume stop: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[kubeexec] ERROR 2018/06/08 09:42:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume delete vol_9c85de66c12db0a72b4d16fe888ff74d] on glusterfs-storage-gxp7c: Err[command terminated with exit code 1]: Stdout []: Stderr [volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>]
>[cmdexec] ERROR 2018/06/08 09:42:36 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:153: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[heketi] ERROR 2018/06/08 09:42:36 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:525: Unable to delete volume: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[heketi] ERROR 2018/06/08 09:42:36 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:429: Error executing delete volume: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[heketi] ERROR 2018/06/08 09:42:36 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:1185: Delete Volume Failed: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[asynchttp] INFO 2018/06/08 09:42:36 asynchttp.go:292: Completed job c90347d62327f9658943dbc7cb811e91 in 805.062677ms
>[negroni] Started GET /queue/c90347d62327f9658943dbc7cb811e91
>[negroni] Completed 500 Internal Server Error in 212.155Âµs
>[heketi] INFO 2018/06/08 09:43:14 Starting Node Health Status refresh
>[cmdexec] INFO 2018/06/08 09:43:14 Check Glusterd service status in node dhcp46-187.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 09:43:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:33 UTC; 3h 19min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48fed567_6ae4_11e8_bc4b_005056a5f18a.slice/docker-a506d9aaca4b440d184df40d01e90b2040811c8090bf4c19d892081307b28c0c.scope/system.slice/glusterd.service
>           ââ  433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1228 /usr/sbin/glusterfsd -s 10.70.46.187 --volfile-id heketidbstorage.10.70.46.187.var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.187-var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.pid -S /var/run/gluster/c08898c2e3b0190c62df3a9d4a013c92.socket --brick-name /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.log --xlator-option *-posix.glusterd-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ20535 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/8fb729a0bbd51b204491c603c84acaa4.socket --xlator-option *replicate*.node-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034
>
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Unit glusterd.service entered failed state.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: glusterd.service failed.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:33 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 09:43:14 Periodic health check status: node 278bd6b4e16a8e62ef15aaae22e6abc1 up=true
>[cmdexec] INFO 2018/06/08 09:43:14 Check Glusterd service status in node dhcp46-122.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 09:43:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:42 UTC; 3h 19min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4900c968_6ae4_11e8_bc4b_005056a5f18a.slice/docker-438fb5fb16a82a3e701b28fa1f1ef8623dc37ca06374c604e5781a2b2ab2e06d.scope/system.slice/glusterd.service
>           ââ  433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1106 /usr/sbin/glusterfsd -s 10.70.46.122 --volfile-id heketidbstorage.10.70.46.122.var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.122-var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.pid -S /var/run/gluster/9fa3ac3b12826912287adc88c26ac134.socket --brick-name /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_29e5b36d921cca1b0699871d9727c932/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.log --xlator-option *-posix.glusterd-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ21076 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f8994cdb09351d83e70da08261d99c64.socket --xlator-option *replicate*.node-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0
>
>Jun 08 06:23:40 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:42 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 09:43:14 Periodic health check status: node 70423cc0bcd044fe5ba8bbbd256a3e49 up=true
>[cmdexec] INFO 2018/06/08 09:43:14 Check Glusterd service status in node dhcp47-76.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 09:43:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 07:25:20 UTC; 2h 17min ago
>  Process: 828 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 830 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5cb2299_6aec_11e8_bc4b_005056a5f18a.slice/docker-7f9223abc9a8641cea68f374f4039235515978ee2dab970ac6cbef182c535426.scope/system.slice/glusterd.service
>           ââ  830 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1197 /usr/sbin/glusterfsd -s 10.70.47.76 --volfile-id heketidbstorage.10.70.47.76.var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.76-var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.pid -S /var/run/gluster/b6976f9ddc3abc90e1dfa91c35a7c611.socket --brick-name /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_54c06cdf52c6fe1e3a3f8a9d39cde33d/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.log --xlator-option *-posix.glusterd-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ13734 /usr/sbin/glusterfsd -s 10.70.47.76 --volfile-id vol_bf31af76ef6c54e7e8f24f4d8711cb22.10.70.47.76.var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_c6935061d1c9dbb05f316e3d87080a38-brick -p /var/run/gluster/vols/vol_bf31af76ef6c54e7e8f24f4d8711cb22/10.70.47.76-var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_c6935061d1c9dbb05f316e3d87080a38-brick.pid -S /var/run/gluster/7aabe8187d4b998d498dd5d4edbd2626.socket --brick-name /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_c6935061d1c9dbb05f316e3d87080a38/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_c6935061d1c9dbb05f316e3d87080a38-brick.log --xlator-option *-posix.glusterd-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94 --brick-port 49153 --xlator-option vol_bf31af76ef6c54e7e8f24f4d8711cb22-server.listen-port=49153
>           ââ18989 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/ec468453b11caed5259f5e1abcc65b44.socket --xlator-option *replicate*.node-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94
>
>Jun 08 07:25:17 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 07:25:20 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Jun 08 08:53:58 dhcp47-76.lab.eng.blr.redhat.com glusterd[830]: [2018-06-08 08:53:58.338441] C [rpc-clnt.c:465:rpc_clnt_fill_request_info] 0-management: cannot lookup the saved frame corresponding to xid (103)
>[heketi] INFO 2018/06/08 09:43:14 Periodic health check status: node d942fe6c0ee5691b7cc263968f97b650 up=true
>[heketi] INFO 2018/06/08 09:43:14 Cleaned 0 nodes from health cache
>[negroni] Started DELETE /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 202 Accepted in 31.76546ms
>[asynchttp] INFO 2018/06/08 09:44:35 asynchttp.go:288: Started job ea262c8ac9568e549031800202ecac77
>[heketi] INFO 2018/06/08 09:44:35 Started async operation: Delete Volume
>[negroni] Started GET /queue/ea262c8ac9568e549031800202ecac77
>[negroni] Completed 200 OK in 169.885Âµs
>[kubeexec] DEBUG 2018/06/08 09:44:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script snapshot list vol_5db55c483f6984d9917cfe4b3c8b3cbc --xml
>Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
><cliOutput>
>  <opRet>-1</opRet>
>  <opErrno>30806</opErrno>
>  <opErrstr>Volume (vol_5db55c483f6984d9917cfe4b3c8b3cbc) does not exist</opErrstr>
></cliOutput>
>[kubeexec] ERROR 2018/06/08 09:44:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume stop vol_5db55c483f6984d9917cfe4b3c8b3cbc force] on glusterfs-storage-pg4xc: Err[command terminated with exit code 1]: Stdout []: Stderr [volume stop: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>]
>[cmdexec] ERROR 2018/06/08 09:44:36 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:144: Unable to stop volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume stop: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[kubeexec] ERROR 2018/06/08 09:44:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume delete vol_5db55c483f6984d9917cfe4b3c8b3cbc] on glusterfs-storage-pg4xc: Err[command terminated with exit code 1]: Stdout []: Stderr [volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>]
>[cmdexec] ERROR 2018/06/08 09:44:36 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:153: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[heketi] ERROR 2018/06/08 09:44:36 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:525: Unable to delete volume: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[heketi] ERROR 2018/06/08 09:44:36 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:429: Error executing delete volume: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[asynchttp] INFO 2018/06/08 09:44:36 asynchttp.go:292: Completed job ea262c8ac9568e549031800202ecac77 in 787.844236ms
>[heketi] ERROR 2018/06/08 09:44:36 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:1185: Delete Volume Failed: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[negroni] Started GET /queue/ea262c8ac9568e549031800202ecac77
>[negroni] Completed 500 Internal Server Error in 250.854Âµs
>[negroni] Started DELETE /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[asynchttp] INFO 2018/06/08 09:44:50 asynchttp.go:288: Started job 6428433f8176b0c80ce6d06033f7adff
>[heketi] INFO 2018/06/08 09:44:50 Started async operation: Delete Volume
>[negroni] Completed 202 Accepted in 26.332473ms
>[negroni] Started GET /queue/6428433f8176b0c80ce6d06033f7adff
>[negroni] Completed 200 OK in 177.537Âµs
>[kubeexec] DEBUG 2018/06/08 09:44:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script snapshot list vol_9c85de66c12db0a72b4d16fe888ff74d --xml
>Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
><cliOutput>
>  <opRet>-1</opRet>
>  <opErrno>30806</opErrno>
>  <opErrstr>Volume (vol_9c85de66c12db0a72b4d16fe888ff74d) does not exist</opErrstr>
></cliOutput>
>[kubeexec] ERROR 2018/06/08 09:44:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume stop vol_9c85de66c12db0a72b4d16fe888ff74d force] on glusterfs-storage-gxp7c: Err[command terminated with exit code 1]: Stdout []: Stderr [volume stop: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>]
>[cmdexec] ERROR 2018/06/08 09:44:51 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:144: Unable to stop volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume stop: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[kubeexec] ERROR 2018/06/08 09:44:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume delete vol_9c85de66c12db0a72b4d16fe888ff74d] on glusterfs-storage-gxp7c: Err[command terminated with exit code 1]: Stdout []: Stderr [volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>]
>[cmdexec] ERROR 2018/06/08 09:44:51 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:153: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[heketi] ERROR 2018/06/08 09:44:51 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:525: Unable to delete volume: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[heketi] ERROR 2018/06/08 09:44:51 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:429: Error executing delete volume: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[heketi] ERROR 2018/06/08 09:44:51 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:1185: Delete Volume Failed: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[asynchttp] INFO 2018/06/08 09:44:51 asynchttp.go:292: Completed job 6428433f8176b0c80ce6d06033f7adff in 787.913606ms
>[negroni] Started GET /queue/6428433f8176b0c80ce6d06033f7adff
>[negroni] Completed 500 Internal Server Error in 187.988Âµs
>[heketi] INFO 2018/06/08 09:45:14 Starting Node Health Status refresh
>[cmdexec] INFO 2018/06/08 09:45:14 Check Glusterd service status in node dhcp46-187.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 09:45:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:33 UTC; 3h 21min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48fed567_6ae4_11e8_bc4b_005056a5f18a.slice/docker-a506d9aaca4b440d184df40d01e90b2040811c8090bf4c19d892081307b28c0c.scope/system.slice/glusterd.service
>           ââ  433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1228 /usr/sbin/glusterfsd -s 10.70.46.187 --volfile-id heketidbstorage.10.70.46.187.var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.187-var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.pid -S /var/run/gluster/c08898c2e3b0190c62df3a9d4a013c92.socket --brick-name /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.log --xlator-option *-posix.glusterd-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ20535 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/8fb729a0bbd51b204491c603c84acaa4.socket --xlator-option *replicate*.node-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034
>
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Unit glusterd.service entered failed state.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: glusterd.service failed.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:33 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 09:45:14 Periodic health check status: node 278bd6b4e16a8e62ef15aaae22e6abc1 up=true
>[cmdexec] INFO 2018/06/08 09:45:14 Check Glusterd service status in node dhcp46-122.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 09:45:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:42 UTC; 3h 21min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4900c968_6ae4_11e8_bc4b_005056a5f18a.slice/docker-438fb5fb16a82a3e701b28fa1f1ef8623dc37ca06374c604e5781a2b2ab2e06d.scope/system.slice/glusterd.service
>           ââ  433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1106 /usr/sbin/glusterfsd -s 10.70.46.122 --volfile-id heketidbstorage.10.70.46.122.var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.122-var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.pid -S /var/run/gluster/9fa3ac3b12826912287adc88c26ac134.socket --brick-name /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_29e5b36d921cca1b0699871d9727c932/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.log --xlator-option *-posix.glusterd-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ21076 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f8994cdb09351d83e70da08261d99c64.socket --xlator-option *replicate*.node-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0
>
>Jun 08 06:23:40 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:42 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 09:45:14 Periodic health check status: node 70423cc0bcd044fe5ba8bbbd256a3e49 up=true
>[cmdexec] INFO 2018/06/08 09:45:14 Check Glusterd service status in node dhcp47-76.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 09:45:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 07:25:20 UTC; 2h 19min ago
>  Process: 828 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 830 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5cb2299_6aec_11e8_bc4b_005056a5f18a.slice/docker-7f9223abc9a8641cea68f374f4039235515978ee2dab970ac6cbef182c535426.scope/system.slice/glusterd.service
>           ââ  830 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1197 /usr/sbin/glusterfsd -s 10.70.47.76 --volfile-id heketidbstorage.10.70.47.76.var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.76-var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.pid -S /var/run/gluster/b6976f9ddc3abc90e1dfa91c35a7c611.socket --brick-name /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_54c06cdf52c6fe1e3a3f8a9d39cde33d/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.log --xlator-option *-posix.glusterd-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ13734 /usr/sbin/glusterfsd -s 10.70.47.76 --volfile-id vol_bf31af76ef6c54e7e8f24f4d8711cb22.10.70.47.76.var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_c6935061d1c9dbb05f316e3d87080a38-brick -p /var/run/gluster/vols/vol_bf31af76ef6c54e7e8f24f4d8711cb22/10.70.47.76-var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_c6935061d1c9dbb05f316e3d87080a38-brick.pid -S /var/run/gluster/7aabe8187d4b998d498dd5d4edbd2626.socket --brick-name /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_c6935061d1c9dbb05f316e3d87080a38/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_c6935061d1c9dbb05f316e3d87080a38-brick.log --xlator-option *-posix.glusterd-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94 --brick-port 49153 --xlator-option vol_bf31af76ef6c54e7e8f24f4d8711cb22-server.listen-port=49153
>           ââ18989 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/ec468453b11caed5259f5e1abcc65b44.socket --xlator-option *replicate*.node-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94
>
>Jun 08 07:25:17 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 07:25:20 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Jun 08 08:53:58 dhcp47-76.lab.eng.blr.redhat.com glusterd[830]: [2018-06-08 08:53:58.338441] C [rpc-clnt.c:465:rpc_clnt_fill_request_info] 0-management: cannot lookup the saved frame corresponding to xid (103)
>[heketi] INFO 2018/06/08 09:45:14 Periodic health check status: node d942fe6c0ee5691b7cc263968f97b650 up=true
>[heketi] INFO 2018/06/08 09:45:14 Cleaned 0 nodes from health cache
>[negroni] Started POST /volumes
>[heketi] INFO 2018/06/08 09:46:07 Allocating brick set #0
>[negroni] Completed 202 Accepted in 55.299668ms
>[asynchttp] INFO 2018/06/08 09:46:07 asynchttp.go:288: Started job 89c9429a70d0111e1376276bdec3c860
>[heketi] INFO 2018/06/08 09:46:07 Started async operation: Create Volume
>[negroni] Started GET /queue/89c9429a70d0111e1376276bdec3c860
>[negroni] Completed 200 OK in 140.95Âµs
>[heketi] INFO 2018/06/08 09:46:07 Creating brick c6c4ef98245b2956a8e3481fb27eb337
>[heketi] INFO 2018/06/08 09:46:07 Creating brick af30399c48acd8c5470723926d83d4c5
>[heketi] INFO 2018/06/08 09:46:07 Creating brick 8f286a6ba640556678264621e234d19d
>[kubeexec] DEBUG 2018/06/08 09:46:07 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir -p /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_af30399c48acd8c5470723926d83d4c5
>Result: 
>[kubeexec] DEBUG 2018/06/08 09:46:07 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir -p /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_8f286a6ba640556678264621e234d19d
>Result: 
>[kubeexec] DEBUG 2018/06/08 09:46:07 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir -p /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_c6c4ef98245b2956a8e3481fb27eb337
>Result: 
>[kubeexec] DEBUG 2018/06/08 09:46:08 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvcreate --autobackup=n --poolmetadatasize 8192K --chunksize 256K --size 1048576K --thin vg_3a4297677881963e3f80124971d50eea/tp_8f286a6ba640556678264621e234d19d --virtualsize 1048576K --name brick_8f286a6ba640556678264621e234d19d
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_8f286a6ba640556678264621e234d19d" created.
>[kubeexec] DEBUG 2018/06/08 09:46:08 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvcreate --autobackup=n --poolmetadatasize 8192K --chunksize 256K --size 1048576K --thin vg_d389f0278a774bd7443a09af960961d8/tp_af30399c48acd8c5470723926d83d4c5 --virtualsize 1048576K --name brick_af30399c48acd8c5470723926d83d4c5
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_af30399c48acd8c5470723926d83d4c5" created.
>[kubeexec] DEBUG 2018/06/08 09:46:08 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvcreate --autobackup=n --poolmetadatasize 8192K --chunksize 256K --size 1048576K --thin vg_9394bc70699b006c5460c9f654cf345f/tp_c6c4ef98245b2956a8e3481fb27eb337 --virtualsize 1048576K --name brick_c6c4ef98245b2956a8e3481fb27eb337
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_c6c4ef98245b2956a8e3481fb27eb337" created.
>[kubeexec] DEBUG 2018/06/08 09:46:08 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_8f286a6ba640556678264621e234d19d
>Result: meta-data=/dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_8f286a6ba640556678264621e234d19d isize=512    agcount=8, agsize=32768 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=262144, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 09:46:08 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_c6c4ef98245b2956a8e3481fb27eb337
>Result: meta-data=/dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_c6c4ef98245b2956a8e3481fb27eb337 isize=512    agcount=8, agsize=32768 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=262144, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 09:46:08 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_af30399c48acd8c5470723926d83d4c5
>Result: meta-data=/dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_af30399c48acd8c5470723926d83d4c5 isize=512    agcount=8, agsize=32768 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=262144, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 09:46:08 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: awk "BEGIN {print \"/dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_8f286a6ba640556678264621e234d19d /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_8f286a6ba640556678264621e234d19d xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 09:46:08 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: awk "BEGIN {print \"/dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_c6c4ef98245b2956a8e3481fb27eb337 /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_c6c4ef98245b2956a8e3481fb27eb337 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 09:46:08 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: awk "BEGIN {print \"/dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_af30399c48acd8c5470723926d83d4c5 /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_af30399c48acd8c5470723926d83d4c5 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[negroni] Started GET /queue/89c9429a70d0111e1376276bdec3c860
>[negroni] Completed 200 OK in 122.729Âµs
>[kubeexec] DEBUG 2018/06/08 09:46:08 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_c6c4ef98245b2956a8e3481fb27eb337 /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_c6c4ef98245b2956a8e3481fb27eb337
>Result: 
>[kubeexec] DEBUG 2018/06/08 09:46:08 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_8f286a6ba640556678264621e234d19d /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_8f286a6ba640556678264621e234d19d
>Result: 
>[kubeexec] DEBUG 2018/06/08 09:46:08 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_af30399c48acd8c5470723926d83d4c5 /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_af30399c48acd8c5470723926d83d4c5
>Result: 
>[kubeexec] DEBUG 2018/06/08 09:46:08 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_c6c4ef98245b2956a8e3481fb27eb337/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 09:46:08 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_8f286a6ba640556678264621e234d19d/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 09:46:08 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_af30399c48acd8c5470723926d83d4c5/brick
>Result: 
>[cmdexec] INFO 2018/06/08 09:46:08 Creating volume vol_5e468916a8596cd844231548ccb61fc4 replica 3
>[kubeexec] DEBUG 2018/06/08 09:46:09 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script volume create vol_5e468916a8596cd844231548ccb61fc4 replica 3 10.70.46.122:/var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_8f286a6ba640556678264621e234d19d/brick 10.70.47.76:/var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_c6c4ef98245b2956a8e3481fb27eb337/brick 10.70.46.187:/var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_af30399c48acd8c5470723926d83d4c5/brick
>Result: volume create: vol_5e468916a8596cd844231548ccb61fc4: success: please start the volume to access data
>[negroni] Started GET /queue/89c9429a70d0111e1376276bdec3c860
>[negroni] Completed 200 OK in 163.892Âµs
>[negroni] Started GET /queue/89c9429a70d0111e1376276bdec3c860
>[negroni] Completed 200 OK in 142.503Âµs
>[kubeexec] DEBUG 2018/06/08 09:46:11 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script volume start vol_5e468916a8596cd844231548ccb61fc4
>Result: volume start: vol_5e468916a8596cd844231548ccb61fc4: success
>[heketi] INFO 2018/06/08 09:46:11 Create Volume succeeded
>[asynchttp] INFO 2018/06/08 09:46:11 asynchttp.go:292: Completed job 89c9429a70d0111e1376276bdec3c860 in 3.79732313s
>[negroni] Started GET /queue/89c9429a70d0111e1376276bdec3c860
>[negroni] Completed 303 See Other in 142.507Âµs
>[negroni] Started GET /volumes/5e468916a8596cd844231548ccb61fc4
>[negroni] Completed 200 OK in 5.547411ms
>[negroni] Started GET /clusters
>[negroni] Completed 200 OK in 1.010554ms
>[negroni] Started GET /clusters/0a73c60efdd4673113b668afea101e6d
>[negroni] Completed 200 OK in 489.884Âµs
>[negroni] Started GET /volumes/07ef5105131fa51c35a9007ee213ea7a
>[negroni] Completed 200 OK in 2.821207ms
>[negroni] Started GET /volumes/15e0122e942fc41f80666a3714670682
>[negroni] Completed 200 OK in 1.507181ms
>[negroni] Started GET /volumes/1918777ef3ce84df17c8a114fb89f33e
>[negroni] Completed 200 OK in 2.182215ms
>[negroni] Started GET /volumes/1a828ff2310b778d09ffdadd755dc5ee
>[negroni] Completed 200 OK in 1.562983ms
>[negroni] Started GET /volumes/1ef58be42cf8ea7cf9298cff303e903a
>[negroni] Completed 200 OK in 2.178457ms
>[negroni] Started GET /volumes/21481d8911fe8ec238d97d71c1aa5cb3
>[negroni] Completed 200 OK in 631.992Âµs
>[negroni] Started GET /volumes/226838416791f3286fcacb7e5f1ff59d
>[negroni] Completed 200 OK in 2.025712ms
>[negroni] Started GET /volumes/2bf097c60bd8b38bfcb4327727ca5681
>[negroni] Completed 200 OK in 1.559063ms
>[negroni] Started GET /volumes/337bf2c01bf8c45eec5bab53ad5c2e46
>[negroni] Completed 200 OK in 2.012177ms
>[negroni] Started GET /volumes/43194b98d83e8b61b376ccc54f79333a
>[negroni] Completed 200 OK in 1.019834ms
>[negroni] Started GET /volumes/5d0dfb0ebb846fcd225c890ec9cdb885
>[negroni] Completed 200 OK in 2.146701ms
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 1.485189ms
>[negroni] Started GET /volumes/5e468916a8596cd844231548ccb61fc4
>[negroni] Completed 200 OK in 1.540386ms
>[negroni] Started GET /volumes/7d5a429f821efc8e8fe3f29569732b86
>[negroni] Completed 200 OK in 1.550956ms
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 2.035948ms
>[negroni] Started GET /volumes/8757685f765bbd74556cbf75086c88f6
>[negroni] Completed 200 OK in 568.778Âµs
>[negroni] Started GET /volumes/89ebb1e7eed2ff557488996a1657e75e
>[negroni] Completed 200 OK in 1.141758ms
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 1.386182ms
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 1.571171ms
>[negroni] Started GET /volumes/9f9fb17746d0aad637b132875d2744e5
>[negroni] Completed 200 OK in 989.229Âµs
>[negroni] Started GET /volumes/9fb2830da79dd70d910dad8426dc236f
>[negroni] Completed 200 OK in 604.467Âµs
>[negroni] Started GET /volumes/a6be87754541710c38b420381c76fb8c
>[negroni] Completed 200 OK in 1.400676ms
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 1.55612ms
>[negroni] Started GET /volumes/a8678c97e2708cf6e00aea160a4d46a0
>[negroni] Completed 200 OK in 518.633Âµs
>[negroni] Started GET /volumes/ad1e5849e9566f1bcaa09cfb9c0b96ef
>[negroni] Completed 200 OK in 1.453336ms
>[negroni] Started GET /volumes/aee5088d2304cb95535752ef85f9f392
>[negroni] Completed 200 OK in 594.324Âµs
>[negroni] Started GET /volumes/afef5f4a84abec3ab3e4f2a5bae2db23
>[negroni] Completed 200 OK in 1.310227ms
>[negroni] Started GET /volumes/b7fb8f19c77039982909b8868f7af4cc
>[negroni] Completed 200 OK in 566.924Âµs
>[negroni] Started GET /volumes/b8f78f128f20b61ef032ce9ee5b6481c
>[negroni] Completed 200 OK in 1.118721ms
>[negroni] Started GET /volumes/b99532640a5201d243193159ee762ae4
>[negroni] Completed 200 OK in 881.186Âµs
>[negroni] Started GET /volumes/bf31af76ef6c54e7e8f24f4d8711cb22
>[negroni] Completed 200 OK in 1.121639ms
>[negroni] Started GET /volumes/c1dce5388e8c136a89e4a25e4cc97821
>[negroni] Completed 200 OK in 588.052Âµs
>[negroni] Started GET /volumes/cc8a686464bb4017c91ba7294ee1b091
>[negroni] Completed 200 OK in 1.120988ms
>[negroni] Started GET /volumes/d21ed39ae095af2674175a798a0cb02c
>[negroni] Completed 200 OK in 572.974Âµs
>[negroni] Started GET /volumes/daf13b7f607d1b280c78e909af25a215
>[negroni] Completed 200 OK in 1.052123ms
>[negroni] Started GET /volumes/dc9ab13a25ccbad8262fba92766a31f9
>[negroni] Completed 200 OK in 1.027713ms
>[negroni] Started GET /volumes/dcaf25d0becadd0bb3c732c2c2ca27da
>[negroni] Completed 200 OK in 558.699Âµs
>[negroni] Started GET /volumes/e25438438fd2d50a0b07f26b4bfb338a
>[negroni] Completed 200 OK in 1.070052ms
>[negroni] Started GET /volumes/f73270331b95278f490fd1dfe0b010df
>[negroni] Completed 200 OK in 661.391Âµs
>[negroni] Started GET /nodes/278bd6b4e16a8e62ef15aaae22e6abc1
>[negroni] Completed 200 OK in 4.345761ms
>[negroni] Started GET /nodes/70423cc0bcd044fe5ba8bbbd256a3e49
>[negroni] Completed 200 OK in 4.148076ms
>[negroni] Started GET /nodes/d942fe6c0ee5691b7cc263968f97b650
>[negroni] Completed 200 OK in 3.664459ms
>[negroni] Started POST /devices
>[heketi] INFO 2018/06/08 09:46:12 Adding device /dev/sdf to node 70423cc0bcd044fe5ba8bbbd256a3e49
>[negroni] Completed 202 Accepted in 9.172133ms
>[asynchttp] INFO 2018/06/08 09:46:12 asynchttp.go:288: Started job 79fc2599d8706785b113872a5ec9fdc1
>[negroni] Started GET /queue/79fc2599d8706785b113872a5ec9fdc1
>[negroni] Completed 200 OK in 155.794Âµs
>[kubeexec] DEBUG 2018/06/08 09:46:12 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: pvcreate --metadatasize=128M --dataalignment=256K '/dev/sdf'
>Result:   Physical volume "/dev/sdf" successfully created.
>[kubeexec] DEBUG 2018/06/08 09:46:13 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: vgcreate --autobackup=n vg_f6e4e4c13df4918c0c15e24e24828609 /dev/sdf
>Result:   Volume group "vg_f6e4e4c13df4918c0c15e24e24828609" successfully created
>[negroni] Started GET /queue/79fc2599d8706785b113872a5ec9fdc1
>[negroni] Completed 200 OK in 118.883Âµs
>[kubeexec] DEBUG 2018/06/08 09:46:13 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: vgdisplay -c vg_f6e4e4c13df4918c0c15e24e24828609
>Result:   vg_f6e4e4c13df4918c0c15e24e24828609:r/w:772:-1:0:0:0:-1:0:1:1:104722432:4096:25567:0:25567:mYTw1o-GBnR-HUKF-0DK9-Nvy6-pAg8-DaSKx0
>[cmdexec] DEBUG 2018/06/08 09:46:13 /src/github.com/heketi/heketi/executors/cmdexec/device.go:147: Size of /dev/sdf in dhcp46-122.lab.eng.blr.redhat.com is 104722432
>[heketi] INFO 2018/06/08 09:46:13 Added device /dev/sdf
>[asynchttp] INFO 2018/06/08 09:46:13 asynchttp.go:292: Completed job 79fc2599d8706785b113872a5ec9fdc1 in 1.144119331s
>[negroni] Started GET /queue/79fc2599d8706785b113872a5ec9fdc1
>[negroni] Completed 204 No Content in 180.292Âµs
>[negroni] Started GET /clusters
>[negroni] Completed 200 OK in 3.263757ms
>[negroni] Started GET /clusters/0a73c60efdd4673113b668afea101e6d
>[negroni] Completed 200 OK in 499.333Âµs
>[negroni] Started GET /volumes/07ef5105131fa51c35a9007ee213ea7a
>[negroni] Completed 200 OK in 4.979446ms
>[negroni] Started GET /volumes/15e0122e942fc41f80666a3714670682
>[negroni] Completed 200 OK in 1.869612ms
>[negroni] Started GET /volumes/1918777ef3ce84df17c8a114fb89f33e
>[negroni] Completed 200 OK in 2.879328ms
>[negroni] Started GET /volumes/1a828ff2310b778d09ffdadd755dc5ee
>[negroni] Completed 200 OK in 1.394615ms
>[negroni] Started GET /volumes/1ef58be42cf8ea7cf9298cff303e903a
>[negroni] Completed 200 OK in 2.69488ms
>[negroni] Started GET /volumes/21481d8911fe8ec238d97d71c1aa5cb3
>[negroni] Completed 200 OK in 981.1Âµs
>[negroni] Started GET /volumes/226838416791f3286fcacb7e5f1ff59d
>[negroni] Completed 200 OK in 2.576768ms
>[negroni] Started GET /volumes/2bf097c60bd8b38bfcb4327727ca5681
>[negroni] Completed 200 OK in 2.143052ms
>[negroni] Started GET /volumes/337bf2c01bf8c45eec5bab53ad5c2e46
>[negroni] Completed 200 OK in 2.593174ms
>[negroni] Started GET /volumes/43194b98d83e8b61b376ccc54f79333a
>[negroni] Completed 200 OK in 1.306531ms
>[negroni] Started GET /volumes/5d0dfb0ebb846fcd225c890ec9cdb885
>[negroni] Completed 200 OK in 2.145851ms
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 2.840117ms
>[negroni] Started GET /volumes/5e468916a8596cd844231548ccb61fc4
>[negroni] Completed 200 OK in 2.450625ms
>[negroni] Started GET /volumes/7d5a429f821efc8e8fe3f29569732b86
>[negroni] Completed 200 OK in 1.767407ms
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 2.579905ms
>[negroni] Started GET /volumes/8757685f765bbd74556cbf75086c88f6
>[negroni] Completed 200 OK in 1.070595ms
>[negroni] Started GET /volumes/89ebb1e7eed2ff557488996a1657e75e
>[negroni] Completed 200 OK in 2.009945ms
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 2.533524ms
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 2.105931ms
>[negroni] Started GET /volumes/9f9fb17746d0aad637b132875d2744e5
>[negroni] Completed 200 OK in 1.634921ms
>[negroni] Started GET /volumes/9fb2830da79dd70d910dad8426dc236f
>[negroni] Completed 200 OK in 530.44Âµs
>[negroni] Started GET /volumes/a6be87754541710c38b420381c76fb8c
>[negroni] Completed 200 OK in 2.341443ms
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 1.595538ms
>[negroni] Started GET /volumes/a8678c97e2708cf6e00aea160a4d46a0
>[negroni] Completed 200 OK in 1.024672ms
>[negroni] Started GET /volumes/ad1e5849e9566f1bcaa09cfb9c0b96ef
>[negroni] Completed 200 OK in 1.761094ms
>[negroni] Started GET /volumes/aee5088d2304cb95535752ef85f9f392
>[negroni] Completed 200 OK in 1.021799ms
>[negroni] Started GET /volumes/afef5f4a84abec3ab3e4f2a5bae2db23
>[negroni] Completed 200 OK in 1.065468ms
>[negroni] Started GET /volumes/b7fb8f19c77039982909b8868f7af4cc
>[negroni] Completed 200 OK in 570.97Âµs
>[negroni] Started GET /volumes/b8f78f128f20b61ef032ce9ee5b6481c
>[negroni] Completed 200 OK in 3.571203ms
>[negroni] Started GET /volumes/b99532640a5201d243193159ee762ae4
>[negroni] Completed 200 OK in 567.961Âµs
>[negroni] Started GET /volumes/bf31af76ef6c54e7e8f24f4d8711cb22
>[negroni] Completed 200 OK in 2.452355ms
>[negroni] Started GET /volumes/c1dce5388e8c136a89e4a25e4cc97821
>[negroni] Completed 200 OK in 727.737Âµs
>[negroni] Started GET /volumes/cc8a686464bb4017c91ba7294ee1b091
>[negroni] Completed 200 OK in 2.234857ms
>[negroni] Started GET /volumes/d21ed39ae095af2674175a798a0cb02c
>[negroni] Completed 200 OK in 915.635Âµs
>[negroni] Started GET /volumes/daf13b7f607d1b280c78e909af25a215
>[negroni] Completed 200 OK in 1.058686ms
>[negroni] Started GET /volumes/dc9ab13a25ccbad8262fba92766a31f9
>[negroni] Completed 200 OK in 1.160171ms
>[negroni] Started GET /volumes/dcaf25d0becadd0bb3c732c2c2ca27da
>[negroni] Completed 200 OK in 687.712Âµs
>[negroni] Started GET /volumes/e25438438fd2d50a0b07f26b4bfb338a
>[negroni] Completed 200 OK in 1.263717ms
>[negroni] Started GET /volumes/f73270331b95278f490fd1dfe0b010df
>[negroni] Completed 200 OK in 673.294Âµs
>[negroni] Started GET /nodes/278bd6b4e16a8e62ef15aaae22e6abc1
>[negroni] Completed 200 OK in 7.131237ms
>[negroni] Started GET /nodes/70423cc0bcd044fe5ba8bbbd256a3e49
>[negroni] Completed 200 OK in 4.28294ms
>[negroni] Started GET /nodes/d942fe6c0ee5691b7cc263968f97b650
>[negroni] Completed 200 OK in 4.174437ms
>[negroni] Started POST /devices/f6e4e4c13df4918c0c15e24e24828609/state
>[negroni] Completed 202 Accepted in 412.578Âµs
>[asynchttp] INFO 2018/06/08 09:46:14 asynchttp.go:288: Started job c7e2e278b9c46482dba755d08dec882e
>[negroni] Started GET /queue/c7e2e278b9c46482dba755d08dec882e
>[negroni] Completed 200 OK in 189.448Âµs
>[asynchttp] INFO 2018/06/08 09:46:14 asynchttp.go:292: Completed job c7e2e278b9c46482dba755d08dec882e in 16.999273ms
>[negroni] Started GET /queue/c7e2e278b9c46482dba755d08dec882e
>[negroni] Completed 204 No Content in 137.678Âµs
>[negroni] Started POST /devices/f6e4e4c13df4918c0c15e24e24828609/state
>[negroni] Completed 202 Accepted in 3.838258ms
>[asynchttp] INFO 2018/06/08 09:46:15 asynchttp.go:288: Started job e3b26b41b8396a6085cd5cc3dee0fe8e
>[heketi] INFO 2018/06/08 09:46:15 Running Remove Device
>[negroni] Started GET /queue/e3b26b41b8396a6085cd5cc3dee0fe8e
>[negroni] Completed 200 OK in 147.133Âµs
>[asynchttp] INFO 2018/06/08 09:46:15 asynchttp.go:292: Completed job e3b26b41b8396a6085cd5cc3dee0fe8e in 16.467713ms
>[negroni] Started GET /queue/e3b26b41b8396a6085cd5cc3dee0fe8e
>[negroni] Completed 204 No Content in 229.441Âµs
>[negroni] Started DELETE /devices/f6e4e4c13df4918c0c15e24e24828609
>[heketi] INFO 2018/06/08 09:46:16 Deleting device f6e4e4c13df4918c0c15e24e24828609 on node 70423cc0bcd044fe5ba8bbbd256a3e49
>[negroni] Completed 202 Accepted in 1.703447ms
>[asynchttp] INFO 2018/06/08 09:46:16 asynchttp.go:288: Started job c4965ed118a6d367683cec4c41b33d71
>[negroni] Started GET /queue/c4965ed118a6d367683cec4c41b33d71
>[negroni] Completed 200 OK in 99.843Âµs
>[kubeexec] DEBUG 2018/06/08 09:46:16 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: vgremove vg_f6e4e4c13df4918c0c15e24e24828609
>Result:   Volume group "vg_f6e4e4c13df4918c0c15e24e24828609" successfully removed
>[kubeexec] DEBUG 2018/06/08 09:46:17 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: pvremove '/dev/sdf'
>Result:   Labels on physical volume "/dev/sdf" successfully wiped.
>[kubeexec] ERROR 2018/06/08 09:46:17 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [ls /var/lib/heketi/mounts/vg_f6e4e4c13df4918c0c15e24e24828609] on glusterfs-storage-pg4xc: Err[command terminated with exit code 2]: Stdout []: Stderr [ls: cannot access /var/lib/heketi/mounts/vg_f6e4e4c13df4918c0c15e24e24828609: No such file or directory
>]
>[heketi] INFO 2018/06/08 09:46:17 Deleted node [f6e4e4c13df4918c0c15e24e24828609]
>[asynchttp] INFO 2018/06/08 09:46:17 asynchttp.go:292: Completed job c4965ed118a6d367683cec4c41b33d71 in 818.099975ms
>[negroni] Started GET /queue/c4965ed118a6d367683cec4c41b33d71
>[negroni] Completed 204 No Content in 180.642Âµs
>[negroni] Started GET /clusters
>[negroni] Completed 200 OK in 2.364399ms
>[negroni] Started GET /clusters/0a73c60efdd4673113b668afea101e6d
>[negroni] Completed 200 OK in 261.699Âµs
>[negroni] Started GET /volumes/07ef5105131fa51c35a9007ee213ea7a
>[negroni] Completed 200 OK in 3.449701ms
>[negroni] Started GET /volumes/15e0122e942fc41f80666a3714670682
>[negroni] Completed 200 OK in 1.568599ms
>[negroni] Started GET /volumes/1918777ef3ce84df17c8a114fb89f33e
>[negroni] Completed 200 OK in 2.326236ms
>[negroni] Started GET /volumes/1a828ff2310b778d09ffdadd755dc5ee
>[negroni] Completed 200 OK in 1.484134ms
>[negroni] Started GET /volumes/1ef58be42cf8ea7cf9298cff303e903a
>[negroni] Completed 200 OK in 2.536054ms
>[negroni] Started GET /volumes/21481d8911fe8ec238d97d71c1aa5cb3
>[negroni] Completed 200 OK in 620.071Âµs
>[negroni] Started GET /volumes/226838416791f3286fcacb7e5f1ff59d
>[negroni] Completed 200 OK in 1.630189ms
>[negroni] Started GET /volumes/2bf097c60bd8b38bfcb4327727ca5681
>[negroni] Completed 200 OK in 1.244244ms
>[negroni] Started GET /volumes/337bf2c01bf8c45eec5bab53ad5c2e46
>[negroni] Completed 200 OK in 1.767543ms
>[negroni] Started GET /volumes/43194b98d83e8b61b376ccc54f79333a
>[negroni] Completed 200 OK in 1.014975ms
>[negroni] Started GET /volumes/5d0dfb0ebb846fcd225c890ec9cdb885
>[negroni] Completed 200 OK in 1.495311ms
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 1.950881ms
>[negroni] Started GET /volumes/5e468916a8596cd844231548ccb61fc4
>[negroni] Completed 200 OK in 1.498553ms
>[negroni] Started GET /volumes/7d5a429f821efc8e8fe3f29569732b86
>[negroni] Completed 200 OK in 983.138Âµs
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 1.395604ms
>[negroni] Started GET /volumes/8757685f765bbd74556cbf75086c88f6
>[negroni] Completed 200 OK in 546.023Âµs
>[negroni] Started GET /volumes/89ebb1e7eed2ff557488996a1657e75e
>[negroni] Completed 200 OK in 1.090569ms
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 1.550387ms
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 1.570478ms
>[negroni] Started GET /volumes/9f9fb17746d0aad637b132875d2744e5
>[negroni] Completed 200 OK in 1.506316ms
>[negroni] Started GET /volumes/9fb2830da79dd70d910dad8426dc236f
>[negroni] Completed 200 OK in 575.326Âµs
>[negroni] Started GET /volumes/a6be87754541710c38b420381c76fb8c
>[negroni] Completed 200 OK in 1.898366ms
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 3.546511ms
>[negroni] Started GET /volumes/a8678c97e2708cf6e00aea160a4d46a0
>[negroni] Completed 200 OK in 1.051006ms
>[negroni] Started GET /volumes/ad1e5849e9566f1bcaa09cfb9c0b96ef
>[negroni] Completed 200 OK in 1.63748ms
>[negroni] Started GET /volumes/aee5088d2304cb95535752ef85f9f392
>[negroni] Completed 200 OK in 1.018125ms
>[negroni] Started GET /volumes/afef5f4a84abec3ab3e4f2a5bae2db23
>[negroni] Completed 200 OK in 2.551033ms
>[negroni] Started GET /volumes/b7fb8f19c77039982909b8868f7af4cc
>[negroni] Completed 200 OK in 975.547Âµs
>[negroni] Started GET /volumes/b8f78f128f20b61ef032ce9ee5b6481c
>[negroni] Completed 200 OK in 1.341515ms
>[negroni] Started GET /volumes/b99532640a5201d243193159ee762ae4
>[negroni] Completed 200 OK in 597.472Âµs
>[negroni] Started GET /volumes/bf31af76ef6c54e7e8f24f4d8711cb22
>[negroni] Completed 200 OK in 1.011749ms
>[negroni] Started GET /volumes/c1dce5388e8c136a89e4a25e4cc97821
>[negroni] Completed 200 OK in 556.759Âµs
>[negroni] Started GET /volumes/cc8a686464bb4017c91ba7294ee1b091
>[negroni] Completed 200 OK in 1.131836ms
>[negroni] Started GET /volumes/d21ed39ae095af2674175a798a0cb02c
>[negroni] Completed 200 OK in 996.355Âµs
>[negroni] Started GET /volumes/daf13b7f607d1b280c78e909af25a215
>[negroni] Completed 200 OK in 979.207Âµs
>[negroni] Started GET /volumes/dc9ab13a25ccbad8262fba92766a31f9
>[negroni] Completed 200 OK in 1.600281ms
>[negroni] Started GET /volumes/dcaf25d0becadd0bb3c732c2c2ca27da
>[negroni] Completed 200 OK in 976.322Âµs
>[negroni] Started GET /volumes/e25438438fd2d50a0b07f26b4bfb338a
>[negroni] Completed 200 OK in 1.432427ms
>[negroni] Started GET /volumes/f73270331b95278f490fd1dfe0b010df
>[negroni] Completed 200 OK in 929.041Âµs
>[negroni] Started GET /nodes/278bd6b4e16a8e62ef15aaae22e6abc1
>[negroni] Completed 200 OK in 6.659692ms
>[negroni] Started GET /nodes/70423cc0bcd044fe5ba8bbbd256a3e49
>[negroni] Completed 200 OK in 3.813247ms
>[negroni] Started GET /nodes/d942fe6c0ee5691b7cc263968f97b650
>[negroni] Completed 200 OK in 4.0463ms
>[negroni] Started POST /volumes
>[heketi] INFO 2018/06/08 09:46:18 Allocating brick set #0
>[negroni] Completed 202 Accepted in 23.590211ms
>[asynchttp] INFO 2018/06/08 09:46:18 asynchttp.go:288: Started job 83fd9f1d6abffe2a8c2d47a2a9dc9349
>[heketi] INFO 2018/06/08 09:46:18 Started async operation: Create Volume
>[negroni] Started GET /queue/83fd9f1d6abffe2a8c2d47a2a9dc9349
>[negroni] Completed 200 OK in 110.35Âµs
>[heketi] INFO 2018/06/08 09:46:18 Creating brick 7d65ac61f84807c36a3615033d19a5d6
>[heketi] INFO 2018/06/08 09:46:18 Creating brick 7bb12d9946ceefb336013d020f571cf6
>[heketi] INFO 2018/06/08 09:46:18 Creating brick eaeff3df9462b6b153a88941afeca9a3
>[kubeexec] DEBUG 2018/06/08 09:46:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir -p /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_eaeff3df9462b6b153a88941afeca9a3
>Result: 
>[kubeexec] DEBUG 2018/06/08 09:46:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir -p /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_7bb12d9946ceefb336013d020f571cf6
>Result: 
>[kubeexec] DEBUG 2018/06/08 09:46:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir -p /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_7d65ac61f84807c36a3615033d19a5d6
>Result: 
>[kubeexec] DEBUG 2018/06/08 09:46:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvcreate --autobackup=n --poolmetadatasize 8192K --chunksize 256K --size 1048576K --thin vg_3a4297677881963e3f80124971d50eea/tp_eaeff3df9462b6b153a88941afeca9a3 --virtualsize 1048576K --name brick_eaeff3df9462b6b153a88941afeca9a3
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_eaeff3df9462b6b153a88941afeca9a3" created.
>[kubeexec] DEBUG 2018/06/08 09:46:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvcreate --autobackup=n --poolmetadatasize 8192K --chunksize 256K --size 1048576K --thin vg_b861af9cc67f5bf8d5a8f9e38f56b14e/tp_7bb12d9946ceefb336013d020f571cf6 --virtualsize 1048576K --name brick_7bb12d9946ceefb336013d020f571cf6
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_7bb12d9946ceefb336013d020f571cf6" created.
>[kubeexec] DEBUG 2018/06/08 09:46:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvcreate --autobackup=n --poolmetadatasize 8192K --chunksize 256K --size 1048576K --thin vg_96f1667f2f1ced2c5ef94772922be93b/tp_7d65ac61f84807c36a3615033d19a5d6 --virtualsize 1048576K --name brick_7d65ac61f84807c36a3615033d19a5d6
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_7d65ac61f84807c36a3615033d19a5d6" created.
>[kubeexec] DEBUG 2018/06/08 09:46:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_eaeff3df9462b6b153a88941afeca9a3
>Result: meta-data=/dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_eaeff3df9462b6b153a88941afeca9a3 isize=512    agcount=8, agsize=32768 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=262144, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 09:46:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_7bb12d9946ceefb336013d020f571cf6
>Result: meta-data=/dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_7bb12d9946ceefb336013d020f571cf6 isize=512    agcount=8, agsize=32768 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=262144, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 09:46:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_7d65ac61f84807c36a3615033d19a5d6
>Result: meta-data=/dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_7d65ac61f84807c36a3615033d19a5d6 isize=512    agcount=8, agsize=32768 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=262144, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 09:46:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: awk "BEGIN {print \"/dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_eaeff3df9462b6b153a88941afeca9a3 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_eaeff3df9462b6b153a88941afeca9a3 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 09:46:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: awk "BEGIN {print \"/dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_7bb12d9946ceefb336013d020f571cf6 /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_7bb12d9946ceefb336013d020f571cf6 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 09:46:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: awk "BEGIN {print \"/dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_7d65ac61f84807c36a3615033d19a5d6 /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_7d65ac61f84807c36a3615033d19a5d6 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 09:46:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_eaeff3df9462b6b153a88941afeca9a3 /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_eaeff3df9462b6b153a88941afeca9a3
>Result: 
>[kubeexec] DEBUG 2018/06/08 09:46:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_7bb12d9946ceefb336013d020f571cf6 /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_7bb12d9946ceefb336013d020f571cf6
>Result: 
>[negroni] Started GET /queue/83fd9f1d6abffe2a8c2d47a2a9dc9349
>[negroni] Completed 200 OK in 118.831Âµs
>[kubeexec] DEBUG 2018/06/08 09:46:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_7d65ac61f84807c36a3615033d19a5d6 /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_7d65ac61f84807c36a3615033d19a5d6
>Result: 
>[kubeexec] DEBUG 2018/06/08 09:46:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_eaeff3df9462b6b153a88941afeca9a3/brick
>Result: 
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 6.822273ms
>[negroni] Started GET /volumes/07ef5105131fa51c35a9007ee213ea7a
>[negroni] Completed 200 OK in 1.861715ms
>[negroni] Started GET /volumes/15e0122e942fc41f80666a3714670682
>[negroni] Completed 200 OK in 1.96889ms
>[negroni] Started GET /volumes/1918777ef3ce84df17c8a114fb89f33e
>[negroni] Completed 200 OK in 2.149022ms
>[negroni] Started GET /volumes/1a828ff2310b778d09ffdadd755dc5ee
>[negroni] Completed 200 OK in 2.005926ms
>[negroni] Started GET /volumes/1ef58be42cf8ea7cf9298cff303e903a
>[kubeexec] DEBUG 2018/06/08 09:46:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_7bb12d9946ceefb336013d020f571cf6/brick
>Result: 
>[negroni] Completed 200 OK in 2.626192ms
>[negroni] Started GET /volumes/21481d8911fe8ec238d97d71c1aa5cb3
>[negroni] Completed 200 OK in 555.675Âµs
>[negroni] Started GET /volumes/226838416791f3286fcacb7e5f1ff59d
>[negroni] Completed 200 OK in 1.434678ms
>[negroni] Started GET /volumes/2bf097c60bd8b38bfcb4327727ca5681
>[kubeexec] DEBUG 2018/06/08 09:46:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_7d65ac61f84807c36a3615033d19a5d6/brick
>Result: 
>[cmdexec] INFO 2018/06/08 09:46:19 Creating volume vol_0ea9a5540c7e0eac6dee700f278e79ac replica 3
>[negroni] Completed 200 OK in 2.454009ms
>[negroni] Started GET /volumes/337bf2c01bf8c45eec5bab53ad5c2e46
>[negroni] Completed 200 OK in 1.229778ms
>[negroni] Started GET /volumes/43194b98d83e8b61b376ccc54f79333a
>[negroni] Completed 200 OK in 535.703Âµs
>[negroni] Started GET /volumes/5d0dfb0ebb846fcd225c890ec9cdb885
>[negroni] Completed 200 OK in 1.265394ms
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 1.263281ms
>[negroni] Started GET /volumes/5e468916a8596cd844231548ccb61fc4
>[negroni] Completed 200 OK in 1.547351ms
>[negroni] Started GET /volumes/7d5a429f821efc8e8fe3f29569732b86
>[negroni] Completed 200 OK in 3.102714ms
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 1.281991ms
>[negroni] Started GET /volumes/8757685f765bbd74556cbf75086c88f6
>[negroni] Completed 200 OK in 501.166Âµs
>[negroni] Started GET /volumes/89ebb1e7eed2ff557488996a1657e75e
>[negroni] Completed 200 OK in 1.056443ms
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 982.542Âµs
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 959.923Âµs
>[negroni] Started GET /volumes/9f9fb17746d0aad637b132875d2744e5
>[negroni] Completed 200 OK in 531.702Âµs
>[negroni] Started GET /volumes/9fb2830da79dd70d910dad8426dc236f
>[negroni] Completed 200 OK in 601.335Âµs
>[negroni] Started GET /volumes/a6be87754541710c38b420381c76fb8c
>[negroni] Completed 200 OK in 1.532824ms
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 497.624Âµs
>[negroni] Started GET /volumes/a8678c97e2708cf6e00aea160a4d46a0
>[negroni] Completed 200 OK in 829.415Âµs
>[negroni] Started GET /volumes/ad1e5849e9566f1bcaa09cfb9c0b96ef
>[negroni] Completed 200 OK in 504.136Âµs
>[negroni] Started GET /volumes/aee5088d2304cb95535752ef85f9f392
>[negroni] Completed 200 OK in 1.008875ms
>[negroni] Started GET /volumes/afef5f4a84abec3ab3e4f2a5bae2db23
>[negroni] Completed 200 OK in 829.617Âµs
>[negroni] Started GET /volumes/b7fb8f19c77039982909b8868f7af4cc
>[negroni] Completed 200 OK in 862.123Âµs
>[negroni] Started GET /volumes/b8f78f128f20b61ef032ce9ee5b6481c
>[negroni] Completed 200 OK in 804.264Âµs
>[negroni] Started GET /volumes/b99532640a5201d243193159ee762ae4
>[negroni] Completed 200 OK in 506.644Âµs
>[negroni] Started GET /volumes/bf31af76ef6c54e7e8f24f4d8711cb22
>[negroni] Completed 200 OK in 469.703Âµs
>[negroni] Started GET /volumes/c1dce5388e8c136a89e4a25e4cc97821
>[negroni] Completed 200 OK in 513.717Âµs
>[negroni] Started GET /volumes/cc8a686464bb4017c91ba7294ee1b091
>[negroni] Completed 200 OK in 558.682Âµs
>[negroni] Started GET /volumes/d21ed39ae095af2674175a798a0cb02c
>[negroni] Completed 200 OK in 500.178Âµs
>[negroni] Started GET /volumes/daf13b7f607d1b280c78e909af25a215
>[negroni] Completed 200 OK in 1.837017ms
>[negroni] Started GET /volumes/dc9ab13a25ccbad8262fba92766a31f9
>[negroni] Completed 200 OK in 525.563Âµs
>[negroni] Started GET /volumes/dcaf25d0becadd0bb3c732c2c2ca27da
>[negroni] Completed 200 OK in 507.402Âµs
>[negroni] Started GET /volumes/e25438438fd2d50a0b07f26b4bfb338a
>[negroni] Completed 200 OK in 814.735Âµs
>[negroni] Started GET /volumes/f73270331b95278f490fd1dfe0b010df
>[negroni] Completed 200 OK in 529.836Âµs
>[kubeexec] DEBUG 2018/06/08 09:46:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script volume create vol_0ea9a5540c7e0eac6dee700f278e79ac replica 3 10.70.47.76:/var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_7d65ac61f84807c36a3615033d19a5d6/brick 10.70.46.187:/var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_7bb12d9946ceefb336013d020f571cf6/brick 10.70.46.122:/var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_eaeff3df9462b6b153a88941afeca9a3/brick
>Result: volume create: vol_0ea9a5540c7e0eac6dee700f278e79ac: success: please start the volume to access data
>[negroni] Started GET /queue/83fd9f1d6abffe2a8c2d47a2a9dc9349
>[negroni] Completed 200 OK in 133.461Âµs
>[negroni] Started GET /queue/83fd9f1d6abffe2a8c2d47a2a9dc9349
>[negroni] Completed 200 OK in 141.199Âµs
>[kubeexec] DEBUG 2018/06/08 09:46:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script volume start vol_0ea9a5540c7e0eac6dee700f278e79ac
>Result: volume start: vol_0ea9a5540c7e0eac6dee700f278e79ac: success
>[heketi] INFO 2018/06/08 09:46:21 Create Volume succeeded
>[asynchttp] INFO 2018/06/08 09:46:21 asynchttp.go:292: Completed job 83fd9f1d6abffe2a8c2d47a2a9dc9349 in 3.691364677s
>[negroni] Started GET /queue/83fd9f1d6abffe2a8c2d47a2a9dc9349
>[negroni] Completed 303 See Other in 181.019Âµs
>[negroni] Started GET /volumes/0ea9a5540c7e0eac6dee700f278e79ac
>[negroni] Completed 200 OK in 3.897644ms
>[negroni] Started DELETE /volumes/0ea9a5540c7e0eac6dee700f278e79ac
>[negroni] Completed 202 Accepted in 18.312145ms
>[asynchttp] INFO 2018/06/08 09:46:22 asynchttp.go:288: Started job ea8fa0a98a4c9c258d7b8678033d370a
>[heketi] INFO 2018/06/08 09:46:22 Started async operation: Delete Volume
>[negroni] Started GET /queue/ea8fa0a98a4c9c258d7b8678033d370a
>[negroni] Completed 200 OK in 116.794Âµs
>[kubeexec] DEBUG 2018/06/08 09:46:22 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: gluster --mode=script snapshot list vol_0ea9a5540c7e0eac6dee700f278e79ac --xml
>Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
><cliOutput>
>  <opRet>0</opRet>
>  <opErrno>0</opErrno>
>  <opErrstr/>
>  <snapList>
>    <count>0</count>
>  </snapList>
></cliOutput>
>[negroni] Started GET /queue/ea8fa0a98a4c9c258d7b8678033d370a
>[negroni] Completed 200 OK in 219.549Âµs
>[negroni] Started GET /queue/ea8fa0a98a4c9c258d7b8678033d370a
>[negroni] Completed 200 OK in 162.721Âµs
>[negroni] Started GET /queue/ea8fa0a98a4c9c258d7b8678033d370a
>[negroni] Completed 200 OK in 144.128Âµs
>[kubeexec] DEBUG 2018/06/08 09:46:25 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: gluster --mode=script volume stop vol_0ea9a5540c7e0eac6dee700f278e79ac force
>Result: volume stop: vol_0ea9a5540c7e0eac6dee700f278e79ac: success
>[kubeexec] DEBUG 2018/06/08 09:46:25 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: gluster --mode=script volume delete vol_0ea9a5540c7e0eac6dee700f278e79ac
>Result: volume delete: vol_0ea9a5540c7e0eac6dee700f278e79ac: success
>[heketi] INFO 2018/06/08 09:46:25 Deleting brick 7d65ac61f84807c36a3615033d19a5d6
>[heketi] INFO 2018/06/08 09:46:25 Deleting brick 7bb12d9946ceefb336013d020f571cf6
>[heketi] INFO 2018/06/08 09:46:25 Deleting brick eaeff3df9462b6b153a88941afeca9a3
>[kubeexec] DEBUG 2018/06/08 09:46:26 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mount | grep -w /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_7d65ac61f84807c36a3615033d19a5d6 | cut -d" " -f1
>Result: /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_7d65ac61f84807c36a3615033d19a5d6
>[kubeexec] DEBUG 2018/06/08 09:46:26 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mount | grep -w /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_eaeff3df9462b6b153a88941afeca9a3 | cut -d" " -f1
>Result: /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_eaeff3df9462b6b153a88941afeca9a3
>[kubeexec] DEBUG 2018/06/08 09:46:26 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mount | grep -w /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_7bb12d9946ceefb336013d020f571cf6 | cut -d" " -f1
>Result: /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_7bb12d9946ceefb336013d020f571cf6
>[kubeexec] DEBUG 2018/06/08 09:46:26 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_eaeff3df9462b6b153a88941afeca9a3
>
>Result:   vg_3a4297677881963e3f80124971d50eea/tp_eaeff3df9462b6b153a88941afeca9a3
>[negroni] Started GET /queue/ea8fa0a98a4c9c258d7b8678033d370a
>[negroni] Completed 200 OK in 274.928Âµs
>[kubeexec] DEBUG 2018/06/08 09:46:26 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_7bb12d9946ceefb336013d020f571cf6
>
>Result:   vg_b861af9cc67f5bf8d5a8f9e38f56b14e/tp_7bb12d9946ceefb336013d020f571cf6
>[kubeexec] DEBUG 2018/06/08 09:46:26 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_7d65ac61f84807c36a3615033d19a5d6
>
>Result:   vg_96f1667f2f1ced2c5ef94772922be93b/tp_7d65ac61f84807c36a3615033d19a5d6
>[negroni] Started GET /queue/ea8fa0a98a4c9c258d7b8678033d370a
>[negroni] Completed 200 OK in 250.708Âµs
>[kubeexec] DEBUG 2018/06/08 09:46:27 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: umount /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_eaeff3df9462b6b153a88941afeca9a3
>Result: 
>[kubeexec] DEBUG 2018/06/08 09:46:27 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: umount /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_7bb12d9946ceefb336013d020f571cf6
>Result: 
>[kubeexec] DEBUG 2018/06/08 09:46:27 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: umount /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_7d65ac61f84807c36a3615033d19a5d6
>Result: 
>[negroni] Started GET /queue/ea8fa0a98a4c9c258d7b8678033d370a
>[negroni] Completed 200 OK in 157.967Âµs
>[kubeexec] DEBUG 2018/06/08 09:46:28 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: sed -i.save "/brick_eaeff3df9462b6b153a88941afeca9a3/d" /var/lib/heketi/fstab
>Result: 
>[kubeexec] DEBUG 2018/06/08 09:46:28 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: sed -i.save "/brick_7bb12d9946ceefb336013d020f571cf6/d" /var/lib/heketi/fstab
>Result: 
>[kubeexec] DEBUG 2018/06/08 09:46:28 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: sed -i.save "/brick_7d65ac61f84807c36a3615033d19a5d6/d" /var/lib/heketi/fstab
>Result: 
>[negroni] Started GET /queue/ea8fa0a98a4c9c258d7b8678033d370a
>[negroni] Completed 200 OK in 224.055Âµs
>[kubeexec] DEBUG 2018/06/08 09:46:29 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvremove --autobackup=n -f /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_eaeff3df9462b6b153a88941afeca9a3
>
>Result:   Logical volume "brick_eaeff3df9462b6b153a88941afeca9a3" successfully removed
>[kubeexec] DEBUG 2018/06/08 09:46:30 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvremove --autobackup=n -f /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_7bb12d9946ceefb336013d020f571cf6
>
>Result:   Logical volume "brick_7bb12d9946ceefb336013d020f571cf6" successfully removed
>[negroni] Started GET /queue/ea8fa0a98a4c9c258d7b8678033d370a
>[negroni] Completed 200 OK in 130.391Âµs
>[kubeexec] DEBUG 2018/06/08 09:46:30 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvremove --autobackup=n -f /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_7d65ac61f84807c36a3615033d19a5d6
>
>Result:   Logical volume "brick_7d65ac61f84807c36a3615033d19a5d6" successfully removed
>[kubeexec] DEBUG 2018/06/08 09:46:30 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvs --noheadings --options=thin_count   vg_3a4297677881963e3f80124971d50eea/tp_eaeff3df9462b6b153a88941afeca9a3
>
>Result:        0
>[kubeexec] DEBUG 2018/06/08 09:46:31 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvs --noheadings --options=thin_count   vg_b861af9cc67f5bf8d5a8f9e38f56b14e/tp_7bb12d9946ceefb336013d020f571cf6
>
>Result:        0
>[negroni] Started GET /queue/ea8fa0a98a4c9c258d7b8678033d370a
>[negroni] Completed 200 OK in 132.589Âµs
>[kubeexec] DEBUG 2018/06/08 09:46:31 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvs --noheadings --options=thin_count   vg_96f1667f2f1ced2c5ef94772922be93b/tp_7d65ac61f84807c36a3615033d19a5d6
>
>Result:        0
>[kubeexec] DEBUG 2018/06/08 09:46:32 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvremove --autobackup=n -f   vg_3a4297677881963e3f80124971d50eea/tp_eaeff3df9462b6b153a88941afeca9a3
>
>Result:   Logical volume "tp_eaeff3df9462b6b153a88941afeca9a3" successfully removed
>[negroni] Started GET /queue/ea8fa0a98a4c9c258d7b8678033d370a
>[negroni] Completed 200 OK in 154.623Âµs
>[kubeexec] DEBUG 2018/06/08 09:46:32 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvremove --autobackup=n -f   vg_b861af9cc67f5bf8d5a8f9e38f56b14e/tp_7bb12d9946ceefb336013d020f571cf6
>
>Result:   Logical volume "tp_7bb12d9946ceefb336013d020f571cf6" successfully removed
>[kubeexec] DEBUG 2018/06/08 09:46:32 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvremove --autobackup=n -f   vg_96f1667f2f1ced2c5ef94772922be93b/tp_7d65ac61f84807c36a3615033d19a5d6
>
>Result:   Logical volume "tp_7d65ac61f84807c36a3615033d19a5d6" successfully removed
>[kubeexec] DEBUG 2018/06/08 09:46:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: rmdir /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_eaeff3df9462b6b153a88941afeca9a3
>Result: 
>[negroni] Started GET /queue/ea8fa0a98a4c9c258d7b8678033d370a
>[negroni] Completed 200 OK in 139.089Âµs
>[kubeexec] DEBUG 2018/06/08 09:46:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: rmdir /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_7bb12d9946ceefb336013d020f571cf6
>Result: 
>[kubeexec] DEBUG 2018/06/08 09:46:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: rmdir /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_7d65ac61f84807c36a3615033d19a5d6
>Result: 
>[heketi] INFO 2018/06/08 09:46:33 Delete Volume succeeded
>[asynchttp] INFO 2018/06/08 09:46:33 asynchttp.go:292: Completed job ea8fa0a98a4c9c258d7b8678033d370a in 11.52867039s
>[negroni] Started GET /queue/ea8fa0a98a4c9c258d7b8678033d370a
>[negroni] Completed 204 No Content in 235.419Âµs
>[negroni] Started POST /devices/f6e4e4c13df4918c0c15e24e24828609/state
>[negroni] Completed 404 Not Found in 2.2765ms
>[negroni] Started DELETE /volumes/5e468916a8596cd844231548ccb61fc4
>[negroni] Completed 202 Accepted in 12.763485ms
>[asynchttp] INFO 2018/06/08 09:46:34 asynchttp.go:288: Started job 9a32fb2da0f6f19e8e275255e90e3ef6
>[heketi] INFO 2018/06/08 09:46:34 Started async operation: Delete Volume
>[negroni] Started GET /queue/9a32fb2da0f6f19e8e275255e90e3ef6
>[negroni] Completed 200 OK in 166.004Âµs
>[kubeexec] DEBUG 2018/06/08 09:46:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script snapshot list vol_5e468916a8596cd844231548ccb61fc4 --xml
>Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
><cliOutput>
>  <opRet>0</opRet>
>  <opErrno>0</opErrno>
>  <opErrstr/>
>  <snapList>
>    <count>0</count>
>  </snapList>
></cliOutput>
>[negroni] Started GET /queue/9a32fb2da0f6f19e8e275255e90e3ef6
>[negroni] Completed 200 OK in 221.052Âµs
>[negroni] Started GET /queue/9a32fb2da0f6f19e8e275255e90e3ef6
>[negroni] Completed 200 OK in 203.811Âµs
>[kubeexec] DEBUG 2018/06/08 09:46:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script volume stop vol_5e468916a8596cd844231548ccb61fc4 force
>Result: volume stop: vol_5e468916a8596cd844231548ccb61fc4: success
>[kubeexec] DEBUG 2018/06/08 09:46:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script volume delete vol_5e468916a8596cd844231548ccb61fc4
>Result: volume delete: vol_5e468916a8596cd844231548ccb61fc4: success
>[heketi] INFO 2018/06/08 09:46:37 Deleting brick 8f286a6ba640556678264621e234d19d
>[heketi] INFO 2018/06/08 09:46:37 Deleting brick af30399c48acd8c5470723926d83d4c5
>[heketi] INFO 2018/06/08 09:46:37 Deleting brick c6c4ef98245b2956a8e3481fb27eb337
>[kubeexec] DEBUG 2018/06/08 09:46:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mount | grep -w /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_c6c4ef98245b2956a8e3481fb27eb337 | cut -d" " -f1
>Result: /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_c6c4ef98245b2956a8e3481fb27eb337
>[kubeexec] DEBUG 2018/06/08 09:46:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mount | grep -w /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_af30399c48acd8c5470723926d83d4c5 | cut -d" " -f1
>Result: /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_af30399c48acd8c5470723926d83d4c5
>[kubeexec] DEBUG 2018/06/08 09:46:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mount | grep -w /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_8f286a6ba640556678264621e234d19d | cut -d" " -f1
>Result: /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_8f286a6ba640556678264621e234d19d
>[kubeexec] DEBUG 2018/06/08 09:46:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_c6c4ef98245b2956a8e3481fb27eb337
>
>Result:   vg_9394bc70699b006c5460c9f654cf345f/tp_c6c4ef98245b2956a8e3481fb27eb337
>[kubeexec] DEBUG 2018/06/08 09:46:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_af30399c48acd8c5470723926d83d4c5
>
>Result:   vg_d389f0278a774bd7443a09af960961d8/tp_af30399c48acd8c5470723926d83d4c5
>[negroni] Started GET /queue/9a32fb2da0f6f19e8e275255e90e3ef6
>[negroni] Completed 200 OK in 125.865Âµs
>[kubeexec] DEBUG 2018/06/08 09:46:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_8f286a6ba640556678264621e234d19d
>
>Result:   vg_3a4297677881963e3f80124971d50eea/tp_8f286a6ba640556678264621e234d19d
>[negroni] Started GET /queue/9a32fb2da0f6f19e8e275255e90e3ef6
>[negroni] Completed 200 OK in 246.774Âµs
>[kubeexec] DEBUG 2018/06/08 09:46:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: umount /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_c6c4ef98245b2956a8e3481fb27eb337
>Result: 
>[kubeexec] DEBUG 2018/06/08 09:46:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: umount /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_af30399c48acd8c5470723926d83d4c5
>Result: 
>[kubeexec] DEBUG 2018/06/08 09:46:39 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: umount /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_8f286a6ba640556678264621e234d19d
>Result: 
>[negroni] Started GET /queue/9a32fb2da0f6f19e8e275255e90e3ef6
>[negroni] Completed 200 OK in 233.172Âµs
>[kubeexec] DEBUG 2018/06/08 09:46:39 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: sed -i.save "/brick_c6c4ef98245b2956a8e3481fb27eb337/d" /var/lib/heketi/fstab
>Result: 
>[kubeexec] DEBUG 2018/06/08 09:46:39 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: sed -i.save "/brick_af30399c48acd8c5470723926d83d4c5/d" /var/lib/heketi/fstab
>Result: 
>[kubeexec] DEBUG 2018/06/08 09:46:40 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: sed -i.save "/brick_8f286a6ba640556678264621e234d19d/d" /var/lib/heketi/fstab
>Result: 
>[negroni] Started GET /queue/9a32fb2da0f6f19e8e275255e90e3ef6
>[negroni] Completed 200 OK in 205.618Âµs
>[kubeexec] DEBUG 2018/06/08 09:46:41 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvremove --autobackup=n -f /dev/mapper/vg_9394bc70699b006c5460c9f654cf345f-brick_c6c4ef98245b2956a8e3481fb27eb337
>
>Result:   Logical volume "brick_c6c4ef98245b2956a8e3481fb27eb337" successfully removed
>[kubeexec] DEBUG 2018/06/08 09:46:41 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvremove --autobackup=n -f /dev/mapper/vg_d389f0278a774bd7443a09af960961d8-brick_af30399c48acd8c5470723926d83d4c5
>
>Result:   Logical volume "brick_af30399c48acd8c5470723926d83d4c5" successfully removed
>[kubeexec] DEBUG 2018/06/08 09:46:41 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvremove --autobackup=n -f /dev/mapper/vg_3a4297677881963e3f80124971d50eea-brick_8f286a6ba640556678264621e234d19d
>
>Result:   Logical volume "brick_8f286a6ba640556678264621e234d19d" successfully removed
>[negroni] Started GET /queue/9a32fb2da0f6f19e8e275255e90e3ef6
>[negroni] Completed 200 OK in 123.932Âµs
>[kubeexec] DEBUG 2018/06/08 09:46:42 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvs --noheadings --options=thin_count   vg_9394bc70699b006c5460c9f654cf345f/tp_c6c4ef98245b2956a8e3481fb27eb337
>
>Result:        0
>[kubeexec] DEBUG 2018/06/08 09:46:42 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvs --noheadings --options=thin_count   vg_d389f0278a774bd7443a09af960961d8/tp_af30399c48acd8c5470723926d83d4c5
>
>Result:        0
>[negroni] Started GET /queue/9a32fb2da0f6f19e8e275255e90e3ef6
>[negroni] Completed 200 OK in 161.236Âµs
>[kubeexec] DEBUG 2018/06/08 09:46:42 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvs --noheadings --options=thin_count   vg_3a4297677881963e3f80124971d50eea/tp_8f286a6ba640556678264621e234d19d
>
>Result:        0
>[kubeexec] DEBUG 2018/06/08 09:46:43 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvremove --autobackup=n -f   vg_9394bc70699b006c5460c9f654cf345f/tp_c6c4ef98245b2956a8e3481fb27eb337
>
>Result:   Logical volume "tp_c6c4ef98245b2956a8e3481fb27eb337" successfully removed
>[negroni] Started GET /queue/9a32fb2da0f6f19e8e275255e90e3ef6
>[negroni] Completed 200 OK in 175.817Âµs
>[kubeexec] DEBUG 2018/06/08 09:46:43 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvremove --autobackup=n -f   vg_d389f0278a774bd7443a09af960961d8/tp_af30399c48acd8c5470723926d83d4c5
>
>Result:   Logical volume "tp_af30399c48acd8c5470723926d83d4c5" successfully removed
>[kubeexec] DEBUG 2018/06/08 09:46:43 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvremove --autobackup=n -f   vg_3a4297677881963e3f80124971d50eea/tp_8f286a6ba640556678264621e234d19d
>
>Result:   Logical volume "tp_8f286a6ba640556678264621e234d19d" successfully removed
>[negroni] Started GET /queue/9a32fb2da0f6f19e8e275255e90e3ef6
>[negroni] Completed 200 OK in 129.444Âµs
>[kubeexec] DEBUG 2018/06/08 09:46:44 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: rmdir /var/lib/heketi/mounts/vg_9394bc70699b006c5460c9f654cf345f/brick_c6c4ef98245b2956a8e3481fb27eb337
>Result: 
>[kubeexec] DEBUG 2018/06/08 09:46:44 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: rmdir /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_af30399c48acd8c5470723926d83d4c5
>Result: 
>[kubeexec] DEBUG 2018/06/08 09:46:44 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: rmdir /var/lib/heketi/mounts/vg_3a4297677881963e3f80124971d50eea/brick_8f286a6ba640556678264621e234d19d
>Result: 
>[heketi] INFO 2018/06/08 09:46:45 Delete Volume succeeded
>[asynchttp] INFO 2018/06/08 09:46:45 asynchttp.go:292: Completed job 9a32fb2da0f6f19e8e275255e90e3ef6 in 10.455956383s
>[negroni] Started GET /queue/9a32fb2da0f6f19e8e275255e90e3ef6
>[negroni] Completed 204 No Content in 176.92Âµs
>[negroni] Started POST /volumes
>[heketi] INFO 2018/06/08 09:46:45 Allocating brick set #0
>[negroni] Completed 202 Accepted in 38.490411ms
>[asynchttp] INFO 2018/06/08 09:46:45 asynchttp.go:288: Started job d64e6cee950d6dc6d84ef4dcbe1e8983
>[heketi] INFO 2018/06/08 09:46:45 Started async operation: Create Volume
>[heketi] INFO 2018/06/08 09:46:45 Creating brick ef4fdbd02d533faf4524683816644ca7
>[heketi] INFO 2018/06/08 09:46:45 Creating brick 626a1c3abdca4c84f7933afe0b14ebd0
>[heketi] INFO 2018/06/08 09:46:45 Creating brick 33dec549534fa9947339ff13b2800c47
>[negroni] Started GET /queue/d64e6cee950d6dc6d84ef4dcbe1e8983
>[negroni] Completed 200 OK in 367.055Âµs
>[kubeexec] DEBUG 2018/06/08 09:46:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir -p /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_626a1c3abdca4c84f7933afe0b14ebd0
>Result: 
>[kubeexec] DEBUG 2018/06/08 09:46:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir -p /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_33dec549534fa9947339ff13b2800c47
>Result: 
>[kubeexec] DEBUG 2018/06/08 09:46:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir -p /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_ef4fdbd02d533faf4524683816644ca7
>Result: 
>[kubeexec] DEBUG 2018/06/08 09:46:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvcreate --autobackup=n --poolmetadatasize 8192K --chunksize 256K --size 1048576K --thin vg_b861af9cc67f5bf8d5a8f9e38f56b14e/tp_626a1c3abdca4c84f7933afe0b14ebd0 --virtualsize 1048576K --name brick_626a1c3abdca4c84f7933afe0b14ebd0
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_626a1c3abdca4c84f7933afe0b14ebd0" created.
>[kubeexec] DEBUG 2018/06/08 09:46:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvcreate --autobackup=n --poolmetadatasize 8192K --chunksize 256K --size 1048576K --thin vg_96f1667f2f1ced2c5ef94772922be93b/tp_33dec549534fa9947339ff13b2800c47 --virtualsize 1048576K --name brick_33dec549534fa9947339ff13b2800c47
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_33dec549534fa9947339ff13b2800c47" created.
>[kubeexec] DEBUG 2018/06/08 09:46:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_626a1c3abdca4c84f7933afe0b14ebd0
>Result: meta-data=/dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_626a1c3abdca4c84f7933afe0b14ebd0 isize=512    agcount=8, agsize=32768 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=262144, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 09:46:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_33dec549534fa9947339ff13b2800c47
>Result: meta-data=/dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_33dec549534fa9947339ff13b2800c47 isize=512    agcount=8, agsize=32768 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=262144, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[kubeexec] DEBUG 2018/06/08 09:46:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: awk "BEGIN {print \"/dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_626a1c3abdca4c84f7933afe0b14ebd0 /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_626a1c3abdca4c84f7933afe0b14ebd0 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 09:46:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvcreate --autobackup=n --poolmetadatasize 8192K --chunksize 256K --size 1048576K --thin vg_66a9af9f6bac95f9d8d556a2f14c29d3/tp_ef4fdbd02d533faf4524683816644ca7 --virtualsize 1048576K --name brick_ef4fdbd02d533faf4524683816644ca7
>Result:   Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data.
>  Logical volume "brick_ef4fdbd02d533faf4524683816644ca7" created.
>[kubeexec] DEBUG 2018/06/08 09:46:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: awk "BEGIN {print \"/dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_33dec549534fa9947339ff13b2800c47 /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_33dec549534fa9947339ff13b2800c47 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 09:46:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_626a1c3abdca4c84f7933afe0b14ebd0 /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_626a1c3abdca4c84f7933afe0b14ebd0
>Result: 
>[kubeexec] DEBUG 2018/06/08 09:46:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mkdir /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_626a1c3abdca4c84f7933afe0b14ebd0/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 09:46:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_33dec549534fa9947339ff13b2800c47 /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_33dec549534fa9947339ff13b2800c47
>Result: 
>[kubeexec] DEBUG 2018/06/08 09:46:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_ef4fdbd02d533faf4524683816644ca7
>Result: meta-data=/dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_ef4fdbd02d533faf4524683816644ca7 isize=512    agcount=8, agsize=32768 blks
>         =                       sectsz=512   attr=2, projid32bit=1
>         =                       crc=1        finobt=0, sparse=0
>data     =                       bsize=4096   blocks=262144, imaxpct=25
>         =                       sunit=64     swidth=64 blks
>naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
>log      =internal log           bsize=4096   blocks=2560, version=2
>         =                       sectsz=512   sunit=64 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>[negroni] Started GET /queue/d64e6cee950d6dc6d84ef4dcbe1e8983
>[negroni] Completed 200 OK in 180.098Âµs
>[kubeexec] DEBUG 2018/06/08 09:46:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mkdir /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_33dec549534fa9947339ff13b2800c47/brick
>Result: 
>[kubeexec] DEBUG 2018/06/08 09:46:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: awk "BEGIN {print \"/dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_ef4fdbd02d533faf4524683816644ca7 /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_ef4fdbd02d533faf4524683816644ca7 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}"
>Result: 
>[kubeexec] DEBUG 2018/06/08 09:46:47 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_ef4fdbd02d533faf4524683816644ca7 /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_ef4fdbd02d533faf4524683816644ca7
>Result: 
>[kubeexec] DEBUG 2018/06/08 09:46:47 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mkdir /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_ef4fdbd02d533faf4524683816644ca7/brick
>Result: 
>[cmdexec] INFO 2018/06/08 09:46:47 Creating volume vol_643274fea1be16f2f01b0800d6211145 replica 3
>[kubeexec] DEBUG 2018/06/08 09:46:47 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script volume create vol_643274fea1be16f2f01b0800d6211145 replica 3 10.70.46.122:/var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_ef4fdbd02d533faf4524683816644ca7/brick 10.70.47.76:/var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_33dec549534fa9947339ff13b2800c47/brick 10.70.46.187:/var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_626a1c3abdca4c84f7933afe0b14ebd0/brick
>Result: volume create: vol_643274fea1be16f2f01b0800d6211145: success: please start the volume to access data
>[negroni] Started GET /queue/d64e6cee950d6dc6d84ef4dcbe1e8983
>[negroni] Completed 200 OK in 184.697Âµs
>[negroni] Started GET /queue/d64e6cee950d6dc6d84ef4dcbe1e8983
>[negroni] Completed 200 OK in 159.351Âµs
>[kubeexec] DEBUG 2018/06/08 09:46:49 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script volume start vol_643274fea1be16f2f01b0800d6211145
>Result: volume start: vol_643274fea1be16f2f01b0800d6211145: success
>[heketi] INFO 2018/06/08 09:46:49 Create Volume succeeded
>[asynchttp] INFO 2018/06/08 09:46:49 asynchttp.go:292: Completed job d64e6cee950d6dc6d84ef4dcbe1e8983 in 3.896902935s
>[negroni] Started GET /queue/d64e6cee950d6dc6d84ef4dcbe1e8983
>[negroni] Completed 303 See Other in 159.686Âµs
>[negroni] Started GET /volumes/643274fea1be16f2f01b0800d6211145
>[negroni] Completed 200 OK in 4.474424ms
>[negroni] Started DELETE /volumes/643274fea1be16f2f01b0800d6211145
>[negroni] Completed 202 Accepted in 9.284556ms
>[asynchttp] INFO 2018/06/08 09:46:49 asynchttp.go:288: Started job 09cb5cee5e35b446c818941864fc4c7e
>[heketi] INFO 2018/06/08 09:46:49 Started async operation: Delete Volume
>[negroni] Started GET /queue/09cb5cee5e35b446c818941864fc4c7e
>[negroni] Completed 200 OK in 109.53Âµs
>[kubeexec] DEBUG 2018/06/08 09:46:50 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script snapshot list vol_643274fea1be16f2f01b0800d6211145 --xml
>Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
><cliOutput>
>  <opRet>0</opRet>
>  <opErrno>0</opErrno>
>  <opErrstr/>
>  <snapList>
>    <count>0</count>
>  </snapList>
></cliOutput>
>[negroni] Started GET /queue/09cb5cee5e35b446c818941864fc4c7e
>[negroni] Completed 200 OK in 136.973Âµs
>[negroni] Started DELETE /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 202 Accepted in 12.15055ms
>[asynchttp] INFO 2018/06/08 09:46:50 asynchttp.go:288: Started job df498c12a6d45ee13eb90ea777810002
>[heketi] INFO 2018/06/08 09:46:50 Started async operation: Delete Volume
>[negroni] Started GET /queue/df498c12a6d45ee13eb90ea777810002
>[negroni] Completed 200 OK in 150.618Âµs
>[kubeexec] DEBUG 2018/06/08 09:46:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script snapshot list vol_5db55c483f6984d9917cfe4b3c8b3cbc --xml
>Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
><cliOutput>
>  <opRet>-1</opRet>
>  <opErrno>30806</opErrno>
>  <opErrstr>Volume (vol_5db55c483f6984d9917cfe4b3c8b3cbc) does not exist</opErrstr>
></cliOutput>
>[negroni] Started GET /queue/09cb5cee5e35b446c818941864fc4c7e
>[negroni] Completed 200 OK in 194.17Âµs
>[negroni] Started GET /queue/df498c12a6d45ee13eb90ea777810002
>[negroni] Completed 200 OK in 126.283Âµs
>[negroni] Started GET /queue/09cb5cee5e35b446c818941864fc4c7e
>[negroni] Completed 200 OK in 154.328Âµs
>[negroni] Started GET /queue/df498c12a6d45ee13eb90ea777810002
>[negroni] Completed 200 OK in 133.027Âµs
>[kubeexec] ERROR 2018/06/08 09:46:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume stop vol_5db55c483f6984d9917cfe4b3c8b3cbc force] on glusterfs-storage-pg4xc: Err[command terminated with exit code 1]: Stdout []: Stderr [volume stop: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>]
>[cmdexec] ERROR 2018/06/08 09:46:53 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:144: Unable to stop volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume stop: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[kubeexec] DEBUG 2018/06/08 09:46:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script volume stop vol_643274fea1be16f2f01b0800d6211145 force
>Result: volume stop: vol_643274fea1be16f2f01b0800d6211145: success
>[kubeexec] ERROR 2018/06/08 09:46:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume delete vol_5db55c483f6984d9917cfe4b3c8b3cbc] on glusterfs-storage-pg4xc: Err[command terminated with exit code 1]: Stdout []: Stderr [volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>]
>[cmdexec] ERROR 2018/06/08 09:46:53 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:153: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[heketi] ERROR 2018/06/08 09:46:53 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:525: Unable to delete volume: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[heketi] ERROR 2018/06/08 09:46:53 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:429: Error executing delete volume: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[heketi] ERROR 2018/06/08 09:46:53 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:1185: Delete Volume Failed: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[asynchttp] INFO 2018/06/08 09:46:53 asynchttp.go:292: Completed job df498c12a6d45ee13eb90ea777810002 in 2.724129188s
>[kubeexec] DEBUG 2018/06/08 09:46:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script volume delete vol_643274fea1be16f2f01b0800d6211145
>Result: volume delete: vol_643274fea1be16f2f01b0800d6211145: success
>[heketi] INFO 2018/06/08 09:46:53 Deleting brick 33dec549534fa9947339ff13b2800c47
>[heketi] INFO 2018/06/08 09:46:53 Deleting brick ef4fdbd02d533faf4524683816644ca7
>[heketi] INFO 2018/06/08 09:46:53 Deleting brick 626a1c3abdca4c84f7933afe0b14ebd0
>[kubeexec] DEBUG 2018/06/08 09:46:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: mount | grep -w /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_33dec549534fa9947339ff13b2800c47 | cut -d" " -f1
>Result: /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_33dec549534fa9947339ff13b2800c47
>[kubeexec] DEBUG 2018/06/08 09:46:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: mount | grep -w /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_ef4fdbd02d533faf4524683816644ca7 | cut -d" " -f1
>Result: /dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_ef4fdbd02d533faf4524683816644ca7
>[kubeexec] DEBUG 2018/06/08 09:46:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: mount | grep -w /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_626a1c3abdca4c84f7933afe0b14ebd0 | cut -d" " -f1
>Result: /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_626a1c3abdca4c84f7933afe0b14ebd0
>[negroni] Started GET /queue/09cb5cee5e35b446c818941864fc4c7e
>[negroni] Completed 200 OK in 155.967Âµs
>[negroni] Started GET /queue/df498c12a6d45ee13eb90ea777810002
>[negroni] Completed 500 Internal Server Error in 119.555Âµs
>[kubeexec] DEBUG 2018/06/08 09:46:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_33dec549534fa9947339ff13b2800c47
>
>Result:   vg_96f1667f2f1ced2c5ef94772922be93b/tp_33dec549534fa9947339ff13b2800c47
>[kubeexec] DEBUG 2018/06/08 09:46:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_ef4fdbd02d533faf4524683816644ca7
>
>Result:   vg_66a9af9f6bac95f9d8d556a2f14c29d3/tp_ef4fdbd02d533faf4524683816644ca7
>[negroni] Started GET /queue/09cb5cee5e35b446c818941864fc4c7e
>[negroni] Completed 200 OK in 290.059Âµs
>[kubeexec] DEBUG 2018/06/08 09:46:55 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvs --noheadings --separator=/ -ovg_name,pool_lv /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_626a1c3abdca4c84f7933afe0b14ebd0
>
>Result:   vg_b861af9cc67f5bf8d5a8f9e38f56b14e/tp_626a1c3abdca4c84f7933afe0b14ebd0
>[kubeexec] DEBUG 2018/06/08 09:46:55 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: umount /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_33dec549534fa9947339ff13b2800c47
>Result: 
>[kubeexec] DEBUG 2018/06/08 09:46:55 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: umount /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_ef4fdbd02d533faf4524683816644ca7
>Result: 
>[negroni] Started GET /queue/09cb5cee5e35b446c818941864fc4c7e
>[negroni] Completed 200 OK in 164.036Âµs
>[kubeexec] DEBUG 2018/06/08 09:46:56 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: umount /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_626a1c3abdca4c84f7933afe0b14ebd0
>Result: 
>[kubeexec] DEBUG 2018/06/08 09:46:56 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: sed -i.save "/brick_33dec549534fa9947339ff13b2800c47/d" /var/lib/heketi/fstab
>Result: 
>[kubeexec] DEBUG 2018/06/08 09:46:56 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: sed -i.save "/brick_ef4fdbd02d533faf4524683816644ca7/d" /var/lib/heketi/fstab
>Result: 
>[negroni] Started GET /queue/09cb5cee5e35b446c818941864fc4c7e
>[negroni] Completed 200 OK in 256.325Âµs
>[kubeexec] DEBUG 2018/06/08 09:46:57 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: sed -i.save "/brick_626a1c3abdca4c84f7933afe0b14ebd0/d" /var/lib/heketi/fstab
>Result: 
>[negroni] Started DELETE /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 202 Accepted in 19.085366ms
>[asynchttp] INFO 2018/06/08 09:46:57 asynchttp.go:288: Started job 8bdb0827c4209c0fd92fd7cbc9a69130
>[heketi] INFO 2018/06/08 09:46:57 Started async operation: Delete Volume
>[negroni] Started GET /queue/8bdb0827c4209c0fd92fd7cbc9a69130
>[negroni] Completed 200 OK in 103.112Âµs
>[kubeexec] DEBUG 2018/06/08 09:46:57 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvremove --autobackup=n -f /dev/mapper/vg_96f1667f2f1ced2c5ef94772922be93b-brick_33dec549534fa9947339ff13b2800c47
>
>Result:   Logical volume "brick_33dec549534fa9947339ff13b2800c47" successfully removed
>[negroni] Started GET /queue/09cb5cee5e35b446c818941864fc4c7e
>[negroni] Completed 200 OK in 140.473Âµs
>[kubeexec] DEBUG 2018/06/08 09:46:57 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvremove --autobackup=n -f /dev/mapper/vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_ef4fdbd02d533faf4524683816644ca7
>
>Result:   Logical volume "brick_ef4fdbd02d533faf4524683816644ca7" successfully removed
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 7.56993ms
>[negroni] Started GET /volumes/07ef5105131fa51c35a9007ee213ea7a
>[negroni] Completed 200 OK in 1.898154ms
>[negroni] Started GET /volumes/15e0122e942fc41f80666a3714670682
>[negroni] Completed 200 OK in 1.533247ms
>[negroni] Started GET /volumes/1918777ef3ce84df17c8a114fb89f33e
>[negroni] Completed 200 OK in 1.338812ms
>[negroni] Started GET /volumes/1a828ff2310b778d09ffdadd755dc5ee
>[negroni] Completed 200 OK in 1.250222ms
>[negroni] Started GET /volumes/1ef58be42cf8ea7cf9298cff303e903a
>[negroni] Completed 200 OK in 1.571701ms
>[negroni] Started GET /volumes/21481d8911fe8ec238d97d71c1aa5cb3
>[negroni] Completed 200 OK in 823.728Âµs
>[negroni] Started GET /volumes/226838416791f3286fcacb7e5f1ff59d
>[negroni] Completed 200 OK in 1.256992ms
>[negroni] Started GET /volumes/2bf097c60bd8b38bfcb4327727ca5681
>[negroni] Completed 200 OK in 1.489248ms
>[negroni] Started GET /volumes/337bf2c01bf8c45eec5bab53ad5c2e46
>[negroni] Completed 200 OK in 1.467833ms
>[negroni] Started GET /volumes/43194b98d83e8b61b376ccc54f79333a
>[negroni] Completed 200 OK in 1.172019ms
>[negroni] Started GET /volumes/5d0dfb0ebb846fcd225c890ec9cdb885
>[negroni] Completed 200 OK in 1.225199ms
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 1.484004ms
>[negroni] Started GET /volumes/643274fea1be16f2f01b0800d6211145
>[negroni] Completed 200 OK in 549.884Âµs
>[negroni] Started GET /volumes/7d5a429f821efc8e8fe3f29569732b86
>[negroni] Completed 200 OK in 1.207518ms
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 857.996Âµs
>[negroni] Started GET /volumes/8757685f765bbd74556cbf75086c88f6
>[negroni] Completed 200 OK in 801.35Âµs
>[negroni] Started GET /volumes/89ebb1e7eed2ff557488996a1657e75e
>[negroni] Completed 200 OK in 1.128875ms
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 774.983Âµs
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 549.207Âµs
>[negroni] Started GET /volumes/9f9fb17746d0aad637b132875d2744e5
>[negroni] Completed 200 OK in 555.501Âµs
>[negroni] Started GET /volumes/9fb2830da79dd70d910dad8426dc236f
>[negroni] Completed 200 OK in 525.833Âµs
>[negroni] Started GET /volumes/a6be87754541710c38b420381c76fb8c
>[negroni] Completed 200 OK in 1.734357ms
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 638.664Âµs
>[negroni] Started GET /volumes/a8678c97e2708cf6e00aea160a4d46a0
>[negroni] Completed 200 OK in 601.029Âµs
>[negroni] Started GET /volumes/ad1e5849e9566f1bcaa09cfb9c0b96ef
>[negroni] Completed 200 OK in 534.107Âµs
>[negroni] Started GET /volumes/aee5088d2304cb95535752ef85f9f392
>[negroni] Completed 200 OK in 1.037091ms
>[negroni] Started GET /volumes/afef5f4a84abec3ab3e4f2a5bae2db23
>[negroni] Completed 200 OK in 581.634Âµs
>[negroni] Started GET /volumes/b7fb8f19c77039982909b8868f7af4cc
>[negroni] Completed 200 OK in 509.715Âµs
>[negroni] Started GET /volumes/b8f78f128f20b61ef032ce9ee5b6481c
>[negroni] Completed 200 OK in 941.381Âµs
>[negroni] Started GET /volumes/b99532640a5201d243193159ee762ae4
>[negroni] Completed 200 OK in 501.622Âµs
>[negroni] Started GET /volumes/bf31af76ef6c54e7e8f24f4d8711cb22
>[negroni] Completed 200 OK in 511.191Âµs
>[negroni] Started GET /volumes/c1dce5388e8c136a89e4a25e4cc97821
>[negroni] Completed 200 OK in 615.044Âµs
>[negroni] Started GET /volumes/cc8a686464bb4017c91ba7294ee1b091
>[negroni] Completed 200 OK in 577.149Âµs
>[negroni] Started GET /volumes/d21ed39ae095af2674175a798a0cb02c
>[negroni] Completed 200 OK in 520.09Âµs
>[negroni] Started GET /volumes/daf13b7f607d1b280c78e909af25a215
>[negroni] Completed 200 OK in 965.515Âµs
>[negroni] Started GET /volumes/dc9ab13a25ccbad8262fba92766a31f9
>[negroni] Completed 200 OK in 545.744Âµs
>[negroni] Started GET /volumes/dcaf25d0becadd0bb3c732c2c2ca27da
>[negroni] Completed 200 OK in 500.812Âµs
>[negroni] Started GET /volumes/e25438438fd2d50a0b07f26b4bfb338a
>[negroni] Completed 200 OK in 480.268Âµs
>[negroni] Started GET /volumes/f73270331b95278f490fd1dfe0b010df
>[negroni] Completed 200 OK in 507.865Âµs
>[kubeexec] DEBUG 2018/06/08 09:46:58 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvremove --autobackup=n -f /dev/mapper/vg_b861af9cc67f5bf8d5a8f9e38f56b14e-brick_626a1c3abdca4c84f7933afe0b14ebd0
>
>Result:   Logical volume "brick_626a1c3abdca4c84f7933afe0b14ebd0" successfully removed
>[negroni] Started GET /queue/8bdb0827c4209c0fd92fd7cbc9a69130
>[negroni] Completed 200 OK in 115.145Âµs
>[kubeexec] DEBUG 2018/06/08 09:46:58 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script snapshot list vol_9c85de66c12db0a72b4d16fe888ff74d --xml
>Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
><cliOutput>
>  <opRet>-1</opRet>
>  <opErrno>30806</opErrno>
>  <opErrstr>Volume (vol_9c85de66c12db0a72b4d16fe888ff74d) does not exist</opErrstr>
></cliOutput>
>[negroni] Started GET /queue/09cb5cee5e35b446c818941864fc4c7e
>[negroni] Completed 200 OK in 137.344Âµs
>[kubeexec] DEBUG 2018/06/08 09:46:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvs --noheadings --options=thin_count   vg_66a9af9f6bac95f9d8d556a2f14c29d3/tp_ef4fdbd02d533faf4524683816644ca7
>
>Result:        0
>[negroni] Started GET /queue/8bdb0827c4209c0fd92fd7cbc9a69130
>[negroni] Completed 200 OK in 238.673Âµs
>[kubeexec] DEBUG 2018/06/08 09:46:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvs --noheadings --options=thin_count   vg_b861af9cc67f5bf8d5a8f9e38f56b14e/tp_626a1c3abdca4c84f7933afe0b14ebd0
>
>Result:        0
>[negroni] Started GET /queue/09cb5cee5e35b446c818941864fc4c7e
>[negroni] Completed 200 OK in 279.714Âµs
>[kubeexec] DEBUG 2018/06/08 09:47:00 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvs --noheadings --options=thin_count   vg_96f1667f2f1ced2c5ef94772922be93b/tp_33dec549534fa9947339ff13b2800c47
>
>Result:        0
>[negroni] Started GET /queue/8bdb0827c4209c0fd92fd7cbc9a69130
>[negroni] Completed 200 OK in 105.455Âµs
>[kubeexec] DEBUG 2018/06/08 09:47:00 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: lvremove --autobackup=n -f   vg_66a9af9f6bac95f9d8d556a2f14c29d3/tp_ef4fdbd02d533faf4524683816644ca7
>
>Result:   Logical volume "tp_ef4fdbd02d533faf4524683816644ca7" successfully removed
>[kubeexec] DEBUG 2018/06/08 09:47:00 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: lvremove --autobackup=n -f   vg_b861af9cc67f5bf8d5a8f9e38f56b14e/tp_626a1c3abdca4c84f7933afe0b14ebd0
>
>Result:   Logical volume "tp_626a1c3abdca4c84f7933afe0b14ebd0" successfully removed
>[negroni] Started GET /queue/09cb5cee5e35b446c818941864fc4c7e
>[negroni] Completed 200 OK in 138.81Âµs
>[kubeexec] ERROR 2018/06/08 09:47:01 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume stop vol_9c85de66c12db0a72b4d16fe888ff74d force] on glusterfs-storage-gxp7c: Err[command terminated with exit code 1]: Stdout []: Stderr [volume stop: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>]
>[cmdexec] ERROR 2018/06/08 09:47:01 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:144: Unable to stop volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume stop: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[negroni] Started GET /queue/8bdb0827c4209c0fd92fd7cbc9a69130
>[negroni] Completed 200 OK in 101.665Âµs
>[kubeexec] DEBUG 2018/06/08 09:47:01 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: rmdir /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_ef4fdbd02d533faf4524683816644ca7
>Result: 
>[negroni] Started GET /queue/09cb5cee5e35b446c818941864fc4c7e
>[negroni] Completed 200 OK in 144.173Âµs
>[kubeexec] DEBUG 2018/06/08 09:47:02 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: rmdir /var/lib/heketi/mounts/vg_b861af9cc67f5bf8d5a8f9e38f56b14e/brick_626a1c3abdca4c84f7933afe0b14ebd0
>Result: 
>[kubeexec] DEBUG 2018/06/08 09:47:02 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: lvremove --autobackup=n -f   vg_96f1667f2f1ced2c5ef94772922be93b/tp_33dec549534fa9947339ff13b2800c47
>
>Result:   Logical volume "tp_33dec549534fa9947339ff13b2800c47" successfully removed
>[negroni] Started GET /queue/8bdb0827c4209c0fd92fd7cbc9a69130
>[negroni] Completed 200 OK in 138.646Âµs
>[kubeexec] ERROR 2018/06/08 09:47:02 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume delete vol_9c85de66c12db0a72b4d16fe888ff74d] on glusterfs-storage-gxp7c: Err[command terminated with exit code 1]: Stdout []: Stderr [volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>]
>[cmdexec] ERROR 2018/06/08 09:47:02 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:153: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[heketi] ERROR 2018/06/08 09:47:02 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:525: Unable to delete volume: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[heketi] ERROR 2018/06/08 09:47:02 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:429: Error executing delete volume: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[asynchttp] INFO 2018/06/08 09:47:02 asynchttp.go:292: Completed job 8bdb0827c4209c0fd92fd7cbc9a69130 in 5.21965916s
>[heketi] ERROR 2018/06/08 09:47:02 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:1185: Delete Volume Failed: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[negroni] Started GET /queue/09cb5cee5e35b446c818941864fc4c7e
>[negroni] Completed 200 OK in 196.283Âµs
>[kubeexec] DEBUG 2018/06/08 09:47:03 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: rmdir /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_33dec549534fa9947339ff13b2800c47
>Result: 
>[heketi] INFO 2018/06/08 09:47:03 Delete Volume succeeded
>[asynchttp] INFO 2018/06/08 09:47:03 asynchttp.go:292: Completed job 09cb5cee5e35b446c818941864fc4c7e in 13.09145499s
>[negroni] Started GET /queue/8bdb0827c4209c0fd92fd7cbc9a69130
>[negroni] Completed 500 Internal Server Error in 166.88Âµs
>[negroni] Started GET /queue/09cb5cee5e35b446c818941864fc4c7e
>[negroni] Completed 204 No Content in 173.293Âµs
>[heketi] INFO 2018/06/08 09:47:14 Starting Node Health Status refresh
>[cmdexec] INFO 2018/06/08 09:47:14 Check Glusterd service status in node dhcp46-187.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 09:47:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:33 UTC; 3h 23min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48fed567_6ae4_11e8_bc4b_005056a5f18a.slice/docker-a506d9aaca4b440d184df40d01e90b2040811c8090bf4c19d892081307b28c0c.scope/system.slice/glusterd.service
>           ââ  433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1228 /usr/sbin/glusterfsd -s 10.70.46.187 --volfile-id heketidbstorage.10.70.46.187.var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.187-var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.pid -S /var/run/gluster/c08898c2e3b0190c62df3a9d4a013c92.socket --brick-name /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.log --xlator-option *-posix.glusterd-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ22979 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/8fb729a0bbd51b204491c603c84acaa4.socket --xlator-option *replicate*.node-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034
>
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Unit glusterd.service entered failed state.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: glusterd.service failed.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:33 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 09:47:14 Periodic health check status: node 278bd6b4e16a8e62ef15aaae22e6abc1 up=true
>[cmdexec] INFO 2018/06/08 09:47:14 Check Glusterd service status in node dhcp46-122.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 09:47:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:42 UTC; 3h 23min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4900c968_6ae4_11e8_bc4b_005056a5f18a.slice/docker-438fb5fb16a82a3e701b28fa1f1ef8623dc37ca06374c604e5781a2b2ab2e06d.scope/system.slice/glusterd.service
>           ââ  433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1106 /usr/sbin/glusterfsd -s 10.70.46.122 --volfile-id heketidbstorage.10.70.46.122.var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.122-var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.pid -S /var/run/gluster/9fa3ac3b12826912287adc88c26ac134.socket --brick-name /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_29e5b36d921cca1b0699871d9727c932/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.log --xlator-option *-posix.glusterd-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ24728 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f8994cdb09351d83e70da08261d99c64.socket --xlator-option *replicate*.node-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0
>
>Jun 08 06:23:40 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:42 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 09:47:14 Periodic health check status: node 70423cc0bcd044fe5ba8bbbd256a3e49 up=true
>[cmdexec] INFO 2018/06/08 09:47:14 Check Glusterd service status in node dhcp47-76.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 09:47:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 07:25:20 UTC; 2h 21min ago
>  Process: 828 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 830 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5cb2299_6aec_11e8_bc4b_005056a5f18a.slice/docker-7f9223abc9a8641cea68f374f4039235515978ee2dab970ac6cbef182c535426.scope/system.slice/glusterd.service
>           ââ  830 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1197 /usr/sbin/glusterfsd -s 10.70.47.76 --volfile-id heketidbstorage.10.70.47.76.var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.76-var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.pid -S /var/run/gluster/b6976f9ddc3abc90e1dfa91c35a7c611.socket --brick-name /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_54c06cdf52c6fe1e3a3f8a9d39cde33d/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.log --xlator-option *-posix.glusterd-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ13734 /usr/sbin/glusterfsd -s 10.70.47.76 --volfile-id vol_bf31af76ef6c54e7e8f24f4d8711cb22.10.70.47.76.var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_c6935061d1c9dbb05f316e3d87080a38-brick -p /var/run/gluster/vols/vol_bf31af76ef6c54e7e8f24f4d8711cb22/10.70.47.76-var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_c6935061d1c9dbb05f316e3d87080a38-brick.pid -S /var/run/gluster/7aabe8187d4b998d498dd5d4edbd2626.socket --brick-name /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_c6935061d1c9dbb05f316e3d87080a38/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_c6935061d1c9dbb05f316e3d87080a38-brick.log --xlator-option *-posix.glusterd-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94 --brick-port 49153 --xlator-option vol_bf31af76ef6c54e7e8f24f4d8711cb22-server.listen-port=49153
>           ââ22654 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/ec468453b11caed5259f5e1abcc65b44.socket --xlator-option *replicate*.node-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94
>
>Jun 08 07:25:17 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 07:25:20 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Jun 08 08:53:58 dhcp47-76.lab.eng.blr.redhat.com glusterd[830]: [2018-06-08 08:53:58.338441] C [rpc-clnt.c:465:rpc_clnt_fill_request_info] 0-management: cannot lookup the saved frame corresponding to xid (103)
>[heketi] INFO 2018/06/08 09:47:14 Periodic health check status: node d942fe6c0ee5691b7cc263968f97b650 up=true
>[heketi] INFO 2018/06/08 09:47:14 Cleaned 0 nodes from health cache
>[negroni] Started DELETE /volumes/vol_a8678c97e2708cf6e00aea160a4d46a0
>[heketi] WARNING 2018/06/08 09:47:36 Invalid path or request /volumes/vol_a8678c97e2708cf6e00aea160a4d46a0
>[negroni] Completed 404 Not Found in 300.201Âµs
>[negroni] Started DELETE /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Started DELETE /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 202 Accepted in 27.100325ms
>[asynchttp] INFO 2018/06/08 09:49:06 asynchttp.go:288: Started job e7e5e1abac474812ec612c70e75804fb
>[heketi] INFO 2018/06/08 09:49:06 Started async operation: Delete Volume
>[negroni] Started GET /queue/e7e5e1abac474812ec612c70e75804fb
>[negroni] Completed 200 OK in 163.978Âµs
>[negroni] Completed 202 Accepted in 50.173801ms
>[asynchttp] INFO 2018/06/08 09:49:06 asynchttp.go:288: Started job 5af0c4eec3041a9f85e5ad13ea51c801
>[heketi] INFO 2018/06/08 09:49:06 Started async operation: Delete Volume
>[negroni] Started GET /queue/5af0c4eec3041a9f85e5ad13ea51c801
>[negroni] Completed 200 OK in 126.372Âµs
>[kubeexec] DEBUG 2018/06/08 09:49:06 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script snapshot list vol_5db55c483f6984d9917cfe4b3c8b3cbc --xml
>Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
><cliOutput>
>  <opRet>-1</opRet>
>  <opErrno>30806</opErrno>
>  <opErrstr>Volume (vol_5db55c483f6984d9917cfe4b3c8b3cbc) does not exist</opErrstr>
></cliOutput>
>[kubeexec] DEBUG 2018/06/08 09:49:06 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script snapshot list vol_9c85de66c12db0a72b4d16fe888ff74d --xml
>Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
><cliOutput>
>  <opRet>-1</opRet>
>  <opErrno>30806</opErrno>
>  <opErrstr>Volume (vol_9c85de66c12db0a72b4d16fe888ff74d) does not exist</opErrstr>
></cliOutput>
>[kubeexec] ERROR 2018/06/08 09:49:06 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume stop vol_5db55c483f6984d9917cfe4b3c8b3cbc force] on glusterfs-storage-pg4xc: Err[command terminated with exit code 1]: Stdout []: Stderr [volume stop: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>]
>[cmdexec] ERROR 2018/06/08 09:49:06 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:144: Unable to stop volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume stop: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[kubeexec] ERROR 2018/06/08 09:49:06 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume stop vol_9c85de66c12db0a72b4d16fe888ff74d force] on glusterfs-storage-gxp7c: Err[command terminated with exit code 1]: Stdout []: Stderr [volume stop: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>]
>[cmdexec] ERROR 2018/06/08 09:49:06 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:144: Unable to stop volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume stop: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[kubeexec] ERROR 2018/06/08 09:49:06 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume delete vol_5db55c483f6984d9917cfe4b3c8b3cbc] on glusterfs-storage-pg4xc: Err[command terminated with exit code 1]: Stdout []: Stderr [volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>]
>[cmdexec] ERROR 2018/06/08 09:49:06 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:153: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[heketi] ERROR 2018/06/08 09:49:06 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:525: Unable to delete volume: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[heketi] ERROR 2018/06/08 09:49:06 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:429: Error executing delete volume: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[kubeexec] ERROR 2018/06/08 09:49:06 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume delete vol_9c85de66c12db0a72b4d16fe888ff74d] on glusterfs-storage-gxp7c: Err[command terminated with exit code 1]: Stdout []: Stderr [volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>]
>[cmdexec] ERROR 2018/06/08 09:49:06 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:153: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[heketi] ERROR 2018/06/08 09:49:06 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:525: Unable to delete volume: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[heketi] ERROR 2018/06/08 09:49:06 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:429: Error executing delete volume: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[heketi] ERROR 2018/06/08 09:49:06 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:1185: Delete Volume Failed: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[asynchttp] INFO 2018/06/08 09:49:06 asynchttp.go:292: Completed job e7e5e1abac474812ec612c70e75804fb in 781.891459ms
>[asynchttp] INFO 2018/06/08 09:49:06 asynchttp.go:292: Completed job 5af0c4eec3041a9f85e5ad13ea51c801 in 772.64938ms
>[heketi] ERROR 2018/06/08 09:49:06 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:1185: Delete Volume Failed: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[negroni] Started GET /queue/e7e5e1abac474812ec612c70e75804fb
>[negroni] Completed 500 Internal Server Error in 158.446Âµs
>[negroni] Started GET /queue/5af0c4eec3041a9f85e5ad13ea51c801
>[negroni] Completed 500 Internal Server Error in 186.129Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 11.789289ms
>[negroni] Started GET /volumes/07ef5105131fa51c35a9007ee213ea7a
>[negroni] Completed 200 OK in 3.25035ms
>[negroni] Started GET /volumes/15e0122e942fc41f80666a3714670682
>[negroni] Completed 200 OK in 2.228214ms
>[negroni] Started GET /volumes/1918777ef3ce84df17c8a114fb89f33e
>[negroni] Completed 200 OK in 2.375436ms
>[negroni] Started GET /volumes/1a828ff2310b778d09ffdadd755dc5ee
>[negroni] Completed 200 OK in 1.867887ms
>[negroni] Started GET /volumes/1ef58be42cf8ea7cf9298cff303e903a
>[negroni] Completed 200 OK in 1.845809ms
>[negroni] Started GET /volumes/21481d8911fe8ec238d97d71c1aa5cb3
>[negroni] Completed 200 OK in 972.409Âµs
>[negroni] Started GET /volumes/226838416791f3286fcacb7e5f1ff59d
>[negroni] Completed 200 OK in 1.773314ms
>[negroni] Started GET /volumes/2bf097c60bd8b38bfcb4327727ca5681
>[negroni] Completed 200 OK in 1.896ms
>[negroni] Started GET /volumes/337bf2c01bf8c45eec5bab53ad5c2e46
>[negroni] Completed 200 OK in 1.873472ms
>[negroni] Started GET /volumes/43194b98d83e8b61b376ccc54f79333a
>[negroni] Completed 200 OK in 1.24484ms
>[negroni] Started GET /volumes/5d0dfb0ebb846fcd225c890ec9cdb885
>[negroni] Completed 200 OK in 1.788782ms
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 1.931802ms
>[negroni] Started GET /volumes/7d5a429f821efc8e8fe3f29569732b86
>[negroni] Completed 200 OK in 1.414164ms
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 1.150463ms
>[negroni] Started GET /volumes/8757685f765bbd74556cbf75086c88f6
>[negroni] Completed 200 OK in 1.071672ms
>[negroni] Started GET /volumes/89ebb1e7eed2ff557488996a1657e75e
>[negroni] Completed 200 OK in 1.389358ms
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 1.884792ms
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 2.192383ms
>[negroni] Started GET /volumes/9f9fb17746d0aad637b132875d2744e5
>[negroni] Completed 200 OK in 917.978Âµs
>[negroni] Started GET /volumes/9fb2830da79dd70d910dad8426dc236f
>[negroni] Completed 200 OK in 751.522Âµs
>[negroni] Started GET /volumes/a6be87754541710c38b420381c76fb8c
>[negroni] Completed 200 OK in 2.879309ms
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 1.235781ms
>[negroni] Started GET /volumes/a8678c97e2708cf6e00aea160a4d46a0
>[negroni] Completed 200 OK in 959.558Âµs
>[negroni] Started GET /volumes/ad1e5849e9566f1bcaa09cfb9c0b96ef
>[negroni] Completed 200 OK in 877.982Âµs
>[negroni] Started GET /volumes/aee5088d2304cb95535752ef85f9f392
>[negroni] Completed 200 OK in 1.702973ms
>[negroni] Started GET /volumes/afef5f4a84abec3ab3e4f2a5bae2db23
>[negroni] Completed 200 OK in 618.237Âµs
>[negroni] Started GET /volumes/b7fb8f19c77039982909b8868f7af4cc
>[negroni] Completed 200 OK in 636.048Âµs
>[negroni] Started GET /volumes/b8f78f128f20b61ef032ce9ee5b6481c
>[negroni] Completed 200 OK in 1.295152ms
>[negroni] Started GET /volumes/b99532640a5201d243193159ee762ae4
>[negroni] Completed 200 OK in 673.668Âµs
>[negroni] Started GET /volumes/bf31af76ef6c54e7e8f24f4d8711cb22
>[negroni] Completed 200 OK in 652.896Âµs
>[negroni] Started GET /volumes/c1dce5388e8c136a89e4a25e4cc97821
>[negroni] Completed 200 OK in 667.489Âµs
>[negroni] Started GET /volumes/cc8a686464bb4017c91ba7294ee1b091
>[negroni] Completed 200 OK in 635.14Âµs
>[negroni] Started GET /volumes/d21ed39ae095af2674175a798a0cb02c
>[negroni] Completed 200 OK in 870.208Âµs
>[negroni] Started GET /volumes/daf13b7f607d1b280c78e909af25a215
>[negroni] Completed 200 OK in 1.473562ms
>[negroni] Started GET /volumes/dc9ab13a25ccbad8262fba92766a31f9
>[negroni] Completed 200 OK in 663.517Âµs
>[negroni] Started GET /volumes/dcaf25d0becadd0bb3c732c2c2ca27da
>[negroni] Completed 200 OK in 865.485Âµs
>[negroni] Started GET /volumes/e25438438fd2d50a0b07f26b4bfb338a
>[negroni] Completed 200 OK in 861.116Âµs
>[negroni] Started GET /volumes/f73270331b95278f490fd1dfe0b010df
>[negroni] Completed 200 OK in 732.657Âµs
>[heketi] INFO 2018/06/08 09:49:14 Starting Node Health Status refresh
>[cmdexec] INFO 2018/06/08 09:49:14 Check Glusterd service status in node dhcp46-187.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 09:49:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:33 UTC; 3h 25min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48fed567_6ae4_11e8_bc4b_005056a5f18a.slice/docker-a506d9aaca4b440d184df40d01e90b2040811c8090bf4c19d892081307b28c0c.scope/system.slice/glusterd.service
>           ââ  433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1228 /usr/sbin/glusterfsd -s 10.70.46.187 --volfile-id heketidbstorage.10.70.46.187.var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.187-var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.pid -S /var/run/gluster/c08898c2e3b0190c62df3a9d4a013c92.socket --brick-name /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.log --xlator-option *-posix.glusterd-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ22979 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/8fb729a0bbd51b204491c603c84acaa4.socket --xlator-option *replicate*.node-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034
>
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Unit glusterd.service entered failed state.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: glusterd.service failed.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:33 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 09:49:14 Periodic health check status: node 278bd6b4e16a8e62ef15aaae22e6abc1 up=true
>[cmdexec] INFO 2018/06/08 09:49:14 Check Glusterd service status in node dhcp46-122.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 09:49:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:42 UTC; 3h 25min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4900c968_6ae4_11e8_bc4b_005056a5f18a.slice/docker-438fb5fb16a82a3e701b28fa1f1ef8623dc37ca06374c604e5781a2b2ab2e06d.scope/system.slice/glusterd.service
>           ââ  433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1106 /usr/sbin/glusterfsd -s 10.70.46.122 --volfile-id heketidbstorage.10.70.46.122.var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.122-var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.pid -S /var/run/gluster/9fa3ac3b12826912287adc88c26ac134.socket --brick-name /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_29e5b36d921cca1b0699871d9727c932/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.log --xlator-option *-posix.glusterd-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ24728 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f8994cdb09351d83e70da08261d99c64.socket --xlator-option *replicate*.node-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0
>
>Jun 08 06:23:40 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:42 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 09:49:14 Periodic health check status: node 70423cc0bcd044fe5ba8bbbd256a3e49 up=true
>[cmdexec] INFO 2018/06/08 09:49:14 Check Glusterd service status in node dhcp47-76.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 09:49:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 07:25:20 UTC; 2h 23min ago
>  Process: 828 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 830 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5cb2299_6aec_11e8_bc4b_005056a5f18a.slice/docker-7f9223abc9a8641cea68f374f4039235515978ee2dab970ac6cbef182c535426.scope/system.slice/glusterd.service
>           ââ  830 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1197 /usr/sbin/glusterfsd -s 10.70.47.76 --volfile-id heketidbstorage.10.70.47.76.var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.76-var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.pid -S /var/run/gluster/b6976f9ddc3abc90e1dfa91c35a7c611.socket --brick-name /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_54c06cdf52c6fe1e3a3f8a9d39cde33d/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.log --xlator-option *-posix.glusterd-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ13734 /usr/sbin/glusterfsd -s 10.70.47.76 --volfile-id vol_bf31af76ef6c54e7e8f24f4d8711cb22.10.70.47.76.var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_c6935061d1c9dbb05f316e3d87080a38-brick -p /var/run/gluster/vols/vol_bf31af76ef6c54e7e8f24f4d8711cb22/10.70.47.76-var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_c6935061d1c9dbb05f316e3d87080a38-brick.pid -S /var/run/gluster/7aabe8187d4b998d498dd5d4edbd2626.socket --brick-name /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_c6935061d1c9dbb05f316e3d87080a38/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_c6935061d1c9dbb05f316e3d87080a38-brick.log --xlator-option *-posix.glusterd-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94 --brick-port 49153 --xlator-option vol_bf31af76ef6c54e7e8f24f4d8711cb22-server.listen-port=49153
>           ââ22654 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/ec468453b11caed5259f5e1abcc65b44.socket --xlator-option *replicate*.node-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94
>
>Jun 08 07:25:17 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 07:25:20 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Jun 08 08:53:58 dhcp47-76.lab.eng.blr.redhat.com glusterd[830]: [2018-06-08 08:53:58.338441] C [rpc-clnt.c:465:rpc_clnt_fill_request_info] 0-management: cannot lookup the saved frame corresponding to xid (103)
>[heketi] INFO 2018/06/08 09:49:14 Periodic health check status: node d942fe6c0ee5691b7cc263968f97b650 up=true
>[heketi] INFO 2018/06/08 09:49:14 Cleaned 0 nodes from health cache
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 323.452Âµs
>[negroni] Started GET /volumes/07ef5105131fa51c35a9007ee213ea7a
>[negroni] Completed 200 OK in 715.323Âµs
>[negroni] Started GET /volumes/15e0122e942fc41f80666a3714670682
>[negroni] Completed 200 OK in 1.045896ms
>[negroni] Started GET /volumes/1918777ef3ce84df17c8a114fb89f33e
>[negroni] Completed 200 OK in 641.228Âµs
>[negroni] Started GET /volumes/1a828ff2310b778d09ffdadd755dc5ee
>[negroni] Completed 200 OK in 580.478Âµs
>[negroni] Started GET /volumes/1ef58be42cf8ea7cf9298cff303e903a
>[negroni] Completed 200 OK in 642.616Âµs
>[negroni] Started GET /volumes/21481d8911fe8ec238d97d71c1aa5cb3
>[negroni] Completed 200 OK in 635.816Âµs
>[negroni] Started GET /volumes/226838416791f3286fcacb7e5f1ff59d
>[negroni] Completed 200 OK in 1.041199ms
>[negroni] Started GET /volumes/2bf097c60bd8b38bfcb4327727ca5681
>[negroni] Completed 200 OK in 1.037963ms
>[negroni] Started GET /volumes/337bf2c01bf8c45eec5bab53ad5c2e46
>[negroni] Completed 200 OK in 564.311Âµs
>[negroni] Started GET /volumes/43194b98d83e8b61b376ccc54f79333a
>[negroni] Completed 200 OK in 608.032Âµs
>[negroni] Started GET /volumes/5d0dfb0ebb846fcd225c890ec9cdb885
>[negroni] Completed 200 OK in 600.062Âµs
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 615.907Âµs
>[negroni] Started GET /volumes/7d5a429f821efc8e8fe3f29569732b86
>[negroni] Completed 200 OK in 1.058543ms
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 605.04Âµs
>[negroni] Started GET /volumes/8757685f765bbd74556cbf75086c88f6
>[negroni] Completed 200 OK in 1.034782ms
>[negroni] Started GET /volumes/89ebb1e7eed2ff557488996a1657e75e
>[negroni] Completed 200 OK in 607.03Âµs
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 633.957Âµs
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 948.933Âµs
>[negroni] Started GET /volumes/9f9fb17746d0aad637b132875d2744e5
>[negroni] Completed 200 OK in 1.025472ms
>[negroni] Started GET /volumes/9fb2830da79dd70d910dad8426dc236f
>[negroni] Completed 200 OK in 1.537436ms
>[negroni] Started GET /volumes/a6be87754541710c38b420381c76fb8c
>[negroni] Completed 200 OK in 891.484Âµs
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 1.066324ms
>[negroni] Started GET /volumes/a8678c97e2708cf6e00aea160a4d46a0
>[negroni] Completed 200 OK in 668.218Âµs
>[negroni] Started GET /volumes/ad1e5849e9566f1bcaa09cfb9c0b96ef
>[negroni] Completed 200 OK in 606.162Âµs
>[negroni] Started GET /volumes/aee5088d2304cb95535752ef85f9f392
>[negroni] Completed 200 OK in 601.849Âµs
>[negroni] Started GET /volumes/afef5f4a84abec3ab3e4f2a5bae2db23
>[negroni] Completed 200 OK in 630.418Âµs
>[negroni] Started GET /volumes/b7fb8f19c77039982909b8868f7af4cc
>[negroni] Completed 200 OK in 613.773Âµs
>[negroni] Started GET /volumes/b8f78f128f20b61ef032ce9ee5b6481c
>[negroni] Completed 200 OK in 660.421Âµs
>[negroni] Started GET /volumes/b99532640a5201d243193159ee762ae4
>[negroni] Completed 200 OK in 608.844Âµs
>[negroni] Started GET /volumes/bf31af76ef6c54e7e8f24f4d8711cb22
>[negroni] Completed 200 OK in 612.336Âµs
>[negroni] Started GET /volumes/c1dce5388e8c136a89e4a25e4cc97821
>[negroni] Completed 200 OK in 605.474Âµs
>[negroni] Started GET /volumes/cc8a686464bb4017c91ba7294ee1b091
>[negroni] Completed 200 OK in 584.536Âµs
>[negroni] Started GET /volumes/d21ed39ae095af2674175a798a0cb02c
>[negroni] Completed 200 OK in 775.373Âµs
>[negroni] Started GET /volumes/daf13b7f607d1b280c78e909af25a215
>[negroni] Completed 200 OK in 754.699Âµs
>[negroni] Started GET /volumes/dc9ab13a25ccbad8262fba92766a31f9
>[negroni] Completed 200 OK in 697.176Âµs
>[negroni] Started GET /volumes/dcaf25d0becadd0bb3c732c2c2ca27da
>[negroni] Completed 200 OK in 735.925Âµs
>[negroni] Started GET /volumes/e25438438fd2d50a0b07f26b4bfb338a
>[negroni] Completed 200 OK in 586.822Âµs
>[negroni] Started GET /volumes/f73270331b95278f490fd1dfe0b010df
>[negroni] Completed 200 OK in 614.001Âµs
>[negroni] Started GET /volumes
>[negroni] Completed 200 OK in 254.711Âµs
>[negroni] Started GET /volumes/07ef5105131fa51c35a9007ee213ea7a
>[negroni] Completed 200 OK in 698.57Âµs
>[negroni] Started GET /volumes/15e0122e942fc41f80666a3714670682
>[negroni] Completed 200 OK in 546.377Âµs
>[negroni] Started GET /volumes/1918777ef3ce84df17c8a114fb89f33e
>[negroni] Completed 200 OK in 668.887Âµs
>[negroni] Started GET /volumes/1a828ff2310b778d09ffdadd755dc5ee
>[negroni] Completed 200 OK in 651.281Âµs
>[negroni] Started GET /volumes/1ef58be42cf8ea7cf9298cff303e903a
>[negroni] Completed 200 OK in 1.116412ms
>[negroni] Started GET /volumes/21481d8911fe8ec238d97d71c1aa5cb3
>[negroni] Completed 200 OK in 1.069092ms
>[negroni] Started GET /volumes/226838416791f3286fcacb7e5f1ff59d
>[negroni] Completed 200 OK in 1.720524ms
>[negroni] Started GET /volumes/2bf097c60bd8b38bfcb4327727ca5681
>[negroni] Completed 200 OK in 1.085712ms
>[negroni] Started GET /volumes/337bf2c01bf8c45eec5bab53ad5c2e46
>[negroni] Completed 200 OK in 582.095Âµs
>[negroni] Started GET /volumes/43194b98d83e8b61b376ccc54f79333a
>[negroni] Completed 200 OK in 643.011Âµs
>[negroni] Started GET /volumes/5d0dfb0ebb846fcd225c890ec9cdb885
>[negroni] Completed 200 OK in 608.419Âµs
>[negroni] Started GET /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Completed 200 OK in 581.241Âµs
>[negroni] Started GET /volumes/7d5a429f821efc8e8fe3f29569732b86
>[negroni] Completed 200 OK in 637.855Âµs
>[negroni] Started GET /volumes/82bdce27665c94e646139a23b29e3033
>[negroni] Completed 200 OK in 1.176005ms
>[negroni] Started GET /volumes/8757685f765bbd74556cbf75086c88f6
>[negroni] Completed 200 OK in 1.305075ms
>[negroni] Started GET /volumes/89ebb1e7eed2ff557488996a1657e75e
>[negroni] Completed 200 OK in 694.716Âµs
>[negroni] Started GET /volumes/9c7d5f0f473cce6914804135f0b8ddcd
>[negroni] Completed 200 OK in 1.034189ms
>[negroni] Started GET /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 200 OK in 907.402Âµs
>[negroni] Started GET /volumes/9f9fb17746d0aad637b132875d2744e5
>[negroni] Completed 200 OK in 910.897Âµs
>[negroni] Started GET /volumes/9fb2830da79dd70d910dad8426dc236f
>[negroni] Completed 200 OK in 714.808Âµs
>[negroni] Started GET /volumes/a6be87754541710c38b420381c76fb8c
>[negroni] Completed 200 OK in 1.669598ms
>[negroni] Started GET /volumes/a7c445e486a69e73d8a54e3278593185
>[negroni] Completed 200 OK in 782.228Âµs
>[negroni] Started GET /volumes/a8678c97e2708cf6e00aea160a4d46a0
>[negroni] Completed 200 OK in 931.088Âµs
>[negroni] Started GET /volumes/ad1e5849e9566f1bcaa09cfb9c0b96ef
>[negroni] Completed 200 OK in 1.287347ms
>[negroni] Started GET /volumes/aee5088d2304cb95535752ef85f9f392
>[negroni] Completed 200 OK in 1.016548ms
>[negroni] Started GET /volumes/afef5f4a84abec3ab3e4f2a5bae2db23
>[negroni] Completed 200 OK in 1.032693ms
>[negroni] Started GET /volumes/b7fb8f19c77039982909b8868f7af4cc
>[negroni] Completed 200 OK in 1.052985ms
>[negroni] Started GET /volumes/b8f78f128f20b61ef032ce9ee5b6481c
>[negroni] Completed 200 OK in 1.039019ms
>[negroni] Started GET /volumes/b99532640a5201d243193159ee762ae4
>[negroni] Completed 200 OK in 846.804Âµs
>[negroni] Started GET /volumes/bf31af76ef6c54e7e8f24f4d8711cb22
>[negroni] Completed 200 OK in 732.922Âµs
>[negroni] Started GET /volumes/c1dce5388e8c136a89e4a25e4cc97821
>[negroni] Completed 200 OK in 544.854Âµs
>[negroni] Started GET /volumes/cc8a686464bb4017c91ba7294ee1b091
>[negroni] Completed 200 OK in 576.89Âµs
>[negroni] Started GET /volumes/d21ed39ae095af2674175a798a0cb02c
>[negroni] Completed 200 OK in 620.681Âµs
>[negroni] Started GET /volumes/daf13b7f607d1b280c78e909af25a215
>[negroni] Completed 200 OK in 580.201Âµs
>[negroni] Started GET /volumes/dc9ab13a25ccbad8262fba92766a31f9
>[negroni] Completed 200 OK in 580.165Âµs
>[negroni] Started GET /volumes/dcaf25d0becadd0bb3c732c2c2ca27da
>[negroni] Completed 200 OK in 591.495Âµs
>[negroni] Started GET /volumes/e25438438fd2d50a0b07f26b4bfb338a
>[negroni] Completed 200 OK in 622.42Âµs
>[negroni] Started GET /volumes/f73270331b95278f490fd1dfe0b010df
>[negroni] Completed 200 OK in 574.18Âµs
>[heketi] INFO 2018/06/08 09:51:14 Starting Node Health Status refresh
>[cmdexec] INFO 2018/06/08 09:51:14 Check Glusterd service status in node dhcp46-187.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 09:51:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:33 UTC; 3h 27min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48fed567_6ae4_11e8_bc4b_005056a5f18a.slice/docker-a506d9aaca4b440d184df40d01e90b2040811c8090bf4c19d892081307b28c0c.scope/system.slice/glusterd.service
>           ââ  433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1228 /usr/sbin/glusterfsd -s 10.70.46.187 --volfile-id heketidbstorage.10.70.46.187.var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.187-var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.pid -S /var/run/gluster/c08898c2e3b0190c62df3a9d4a013c92.socket --brick-name /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.log --xlator-option *-posix.glusterd-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ22979 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/8fb729a0bbd51b204491c603c84acaa4.socket --xlator-option *replicate*.node-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034
>
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Unit glusterd.service entered failed state.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: glusterd.service failed.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:33 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 09:51:14 Periodic health check status: node 278bd6b4e16a8e62ef15aaae22e6abc1 up=true
>[cmdexec] INFO 2018/06/08 09:51:14 Check Glusterd service status in node dhcp46-122.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 09:51:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:42 UTC; 3h 27min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4900c968_6ae4_11e8_bc4b_005056a5f18a.slice/docker-438fb5fb16a82a3e701b28fa1f1ef8623dc37ca06374c604e5781a2b2ab2e06d.scope/system.slice/glusterd.service
>           ââ  433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1106 /usr/sbin/glusterfsd -s 10.70.46.122 --volfile-id heketidbstorage.10.70.46.122.var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.122-var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.pid -S /var/run/gluster/9fa3ac3b12826912287adc88c26ac134.socket --brick-name /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_29e5b36d921cca1b0699871d9727c932/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.log --xlator-option *-posix.glusterd-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ24728 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f8994cdb09351d83e70da08261d99c64.socket --xlator-option *replicate*.node-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0
>
>Jun 08 06:23:40 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:42 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 09:51:14 Periodic health check status: node 70423cc0bcd044fe5ba8bbbd256a3e49 up=true
>[cmdexec] INFO 2018/06/08 09:51:14 Check Glusterd service status in node dhcp47-76.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 09:51:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 07:25:20 UTC; 2h 25min ago
>  Process: 828 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 830 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5cb2299_6aec_11e8_bc4b_005056a5f18a.slice/docker-7f9223abc9a8641cea68f374f4039235515978ee2dab970ac6cbef182c535426.scope/system.slice/glusterd.service
>           ââ  830 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1197 /usr/sbin/glusterfsd -s 10.70.47.76 --volfile-id heketidbstorage.10.70.47.76.var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.76-var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.pid -S /var/run/gluster/b6976f9ddc3abc90e1dfa91c35a7c611.socket --brick-name /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_54c06cdf52c6fe1e3a3f8a9d39cde33d/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.log --xlator-option *-posix.glusterd-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ13734 /usr/sbin/glusterfsd -s 10.70.47.76 --volfile-id vol_bf31af76ef6c54e7e8f24f4d8711cb22.10.70.47.76.var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_c6935061d1c9dbb05f316e3d87080a38-brick -p /var/run/gluster/vols/vol_bf31af76ef6c54e7e8f24f4d8711cb22/10.70.47.76-var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_c6935061d1c9dbb05f316e3d87080a38-brick.pid -S /var/run/gluster/7aabe8187d4b998d498dd5d4edbd2626.socket --brick-name /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_c6935061d1c9dbb05f316e3d87080a38/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_c6935061d1c9dbb05f316e3d87080a38-brick.log --xlator-option *-posix.glusterd-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94 --brick-port 49153 --xlator-option vol_bf31af76ef6c54e7e8f24f4d8711cb22-server.listen-port=49153
>           ââ22654 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/ec468453b11caed5259f5e1abcc65b44.socket --xlator-option *replicate*.node-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94
>
>Jun 08 07:25:17 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 07:25:20 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Jun 08 08:53:58 dhcp47-76.lab.eng.blr.redhat.com glusterd[830]: [2018-06-08 08:53:58.338441] C [rpc-clnt.c:465:rpc_clnt_fill_request_info] 0-management: cannot lookup the saved frame corresponding to xid (103)
>[heketi] INFO 2018/06/08 09:51:14 Periodic health check status: node d942fe6c0ee5691b7cc263968f97b650 up=true
>[heketi] INFO 2018/06/08 09:51:14 Cleaned 0 nodes from health cache
>[negroni] Started DELETE /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Started DELETE /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 202 Accepted in 17.638262ms
>[asynchttp] INFO 2018/06/08 09:51:20 asynchttp.go:288: Started job f3ae4c6e96e94eefdfa0162e5f6c7bce
>[heketi] INFO 2018/06/08 09:51:20 Started async operation: Delete Volume
>[negroni] Started GET /queue/f3ae4c6e96e94eefdfa0162e5f6c7bce
>[negroni] Completed 200 OK in 220.329Âµs
>[negroni] Completed 202 Accepted in 43.291398ms
>[asynchttp] INFO 2018/06/08 09:51:21 asynchttp.go:288: Started job 0c8c6b6bbcc0c09db45f0a14863f2c6a
>[heketi] INFO 2018/06/08 09:51:21 Started async operation: Delete Volume
>[negroni] Started GET /queue/0c8c6b6bbcc0c09db45f0a14863f2c6a
>[negroni] Completed 200 OK in 89.435Âµs
>[kubeexec] DEBUG 2018/06/08 09:51:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script snapshot list vol_5db55c483f6984d9917cfe4b3c8b3cbc --xml
>Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
><cliOutput>
>  <opRet>-1</opRet>
>  <opErrno>30806</opErrno>
>  <opErrstr>Volume (vol_5db55c483f6984d9917cfe4b3c8b3cbc) does not exist</opErrstr>
></cliOutput>
>[kubeexec] DEBUG 2018/06/08 09:51:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script snapshot list vol_9c85de66c12db0a72b4d16fe888ff74d --xml
>Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
><cliOutput>
>  <opRet>-1</opRet>
>  <opErrno>30806</opErrno>
>  <opErrstr>Volume (vol_9c85de66c12db0a72b4d16fe888ff74d) does not exist</opErrstr>
></cliOutput>
>[kubeexec] ERROR 2018/06/08 09:51:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume stop vol_5db55c483f6984d9917cfe4b3c8b3cbc force] on glusterfs-storage-pg4xc: Err[command terminated with exit code 1]: Stdout []: Stderr [volume stop: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>]
>[cmdexec] ERROR 2018/06/08 09:51:21 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:144: Unable to stop volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume stop: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[kubeexec] ERROR 2018/06/08 09:51:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume stop vol_9c85de66c12db0a72b4d16fe888ff74d force] on glusterfs-storage-gxp7c: Err[command terminated with exit code 1]: Stdout []: Stderr [volume stop: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>]
>[cmdexec] ERROR 2018/06/08 09:51:21 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:144: Unable to stop volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume stop: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[kubeexec] ERROR 2018/06/08 09:51:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume delete vol_5db55c483f6984d9917cfe4b3c8b3cbc] on glusterfs-storage-pg4xc: Err[command terminated with exit code 1]: Stdout []: Stderr [volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>]
>[cmdexec] ERROR 2018/06/08 09:51:21 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:153: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[heketi] ERROR 2018/06/08 09:51:21 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:525: Unable to delete volume: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[heketi] ERROR 2018/06/08 09:51:21 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:429: Error executing delete volume: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[asynchttp] INFO 2018/06/08 09:51:21 asynchttp.go:292: Completed job f3ae4c6e96e94eefdfa0162e5f6c7bce in 778.183068ms
>[heketi] ERROR 2018/06/08 09:51:21 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:1185: Delete Volume Failed: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[kubeexec] ERROR 2018/06/08 09:51:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume delete vol_9c85de66c12db0a72b4d16fe888ff74d] on glusterfs-storage-gxp7c: Err[command terminated with exit code 1]: Stdout []: Stderr [volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>]
>[cmdexec] ERROR 2018/06/08 09:51:21 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:153: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[heketi] ERROR 2018/06/08 09:51:21 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:525: Unable to delete volume: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[heketi] ERROR 2018/06/08 09:51:21 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:429: Error executing delete volume: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[asynchttp] INFO 2018/06/08 09:51:21 asynchttp.go:292: Completed job 0c8c6b6bbcc0c09db45f0a14863f2c6a in 778.00841ms
>[heketi] ERROR 2018/06/08 09:51:21 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:1185: Delete Volume Failed: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[negroni] Started GET /queue/f3ae4c6e96e94eefdfa0162e5f6c7bce
>[negroni] Completed 500 Internal Server Error in 199.242Âµs
>[negroni] Started GET /queue/0c8c6b6bbcc0c09db45f0a14863f2c6a
>[negroni] Completed 500 Internal Server Error in 210.333Âµs
>[negroni] Started DELETE /volumes/vol_a8678c97e2708cf6e00aea160a4d46a0
>[heketi] WARNING 2018/06/08 09:53:12 Invalid path or request /volumes/vol_a8678c97e2708cf6e00aea160a4d46a0
>[negroni] Completed 404 Not Found in 194.067Âµs
>[heketi] INFO 2018/06/08 09:53:14 Starting Node Health Status refresh
>[cmdexec] INFO 2018/06/08 09:53:14 Check Glusterd service status in node dhcp46-187.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 09:53:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-187.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh2m Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:33 UTC; 3h 29min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48fed567_6ae4_11e8_bc4b_005056a5f18a.slice/docker-a506d9aaca4b440d184df40d01e90b2040811c8090bf4c19d892081307b28c0c.scope/system.slice/glusterd.service
>           ââ  433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1228 /usr/sbin/glusterfsd -s 10.70.46.187 --volfile-id heketidbstorage.10.70.46.187.var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.187-var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.pid -S /var/run/gluster/c08898c2e3b0190c62df3a9d4a013c92.socket --brick-name /var/lib/heketi/mounts/vg_d389f0278a774bd7443a09af960961d8/brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_d389f0278a774bd7443a09af960961d8-brick_a7665c2ce0e7d81c9d3b9e4e2fb0e506-brick.log --xlator-option *-posix.glusterd-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ22979 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/8fb729a0bbd51b204491c603c84acaa4.socket --xlator-option *replicate*.node-uuid=aa138bf5-fef6-494f-884c-a949f7f1f034
>
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Unit glusterd.service entered failed state.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: glusterd.service failed.
>Jun 08 06:23:31 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:33 dhcp46-187.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 09:53:14 Periodic health check status: node 278bd6b4e16a8e62ef15aaae22e6abc1 up=true
>[cmdexec] INFO 2018/06/08 09:53:14 Check Glusterd service status in node dhcp46-122.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 09:53:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 06:23:42 UTC; 3h 29min ago
>  Process: 432 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 433 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4900c968_6ae4_11e8_bc4b_005056a5f18a.slice/docker-438fb5fb16a82a3e701b28fa1f1ef8623dc37ca06374c604e5781a2b2ab2e06d.scope/system.slice/glusterd.service
>           ââ  433 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1106 /usr/sbin/glusterfsd -s 10.70.46.122 --volfile-id heketidbstorage.10.70.46.122.var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.122-var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.pid -S /var/run/gluster/9fa3ac3b12826912287adc88c26ac134.socket --brick-name /var/lib/heketi/mounts/vg_66a9af9f6bac95f9d8d556a2f14c29d3/brick_29e5b36d921cca1b0699871d9727c932/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_66a9af9f6bac95f9d8d556a2f14c29d3-brick_29e5b36d921cca1b0699871d9727c932-brick.log --xlator-option *-posix.glusterd-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ24728 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f8994cdb09351d83e70da08261d99c64.socket --xlator-option *replicate*.node-uuid=8b7c7b5c-3c70-4970-9fb7-e4805e6b65f0
>
>Jun 08 06:23:40 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 06:23:42 dhcp46-122.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>[heketi] INFO 2018/06/08 09:53:14 Periodic health check status: node 70423cc0bcd044fe5ba8bbbd256a3e49 up=true
>[cmdexec] INFO 2018/06/08 09:53:14 Check Glusterd service status in node dhcp47-76.lab.eng.blr.redhat.com
>[kubeexec] DEBUG 2018/06/08 09:53:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: systemctl status glusterd
>Result: â glusterd.service - GlusterFS, a clustered file-system server
>   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
>   Active: active (running) since Fri 2018-06-08 07:25:20 UTC; 2h 27min ago
>  Process: 828 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
> Main PID: 830 (glusterd)
>   CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5cb2299_6aec_11e8_bc4b_005056a5f18a.slice/docker-7f9223abc9a8641cea68f374f4039235515978ee2dab970ac6cbef182c535426.scope/system.slice/glusterd.service
>           ââ  830 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>           ââ 1197 /usr/sbin/glusterfsd -s 10.70.47.76 --volfile-id heketidbstorage.10.70.47.76.var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.76-var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.pid -S /var/run/gluster/b6976f9ddc3abc90e1dfa91c35a7c611.socket --brick-name /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_54c06cdf52c6fe1e3a3f8a9d39cde33d/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_54c06cdf52c6fe1e3a3f8a9d39cde33d-brick.log --xlator-option *-posix.glusterd-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152
>           ââ13734 /usr/sbin/glusterfsd -s 10.70.47.76 --volfile-id vol_bf31af76ef6c54e7e8f24f4d8711cb22.10.70.47.76.var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_c6935061d1c9dbb05f316e3d87080a38-brick -p /var/run/gluster/vols/vol_bf31af76ef6c54e7e8f24f4d8711cb22/10.70.47.76-var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_c6935061d1c9dbb05f316e3d87080a38-brick.pid -S /var/run/gluster/7aabe8187d4b998d498dd5d4edbd2626.socket --brick-name /var/lib/heketi/mounts/vg_96f1667f2f1ced2c5ef94772922be93b/brick_c6935061d1c9dbb05f316e3d87080a38/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_96f1667f2f1ced2c5ef94772922be93b-brick_c6935061d1c9dbb05f316e3d87080a38-brick.log --xlator-option *-posix.glusterd-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94 --brick-port 49153 --xlator-option vol_bf31af76ef6c54e7e8f24f4d8711cb22-server.listen-port=49153
>           ââ22654 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/ec468453b11caed5259f5e1abcc65b44.socket --xlator-option *replicate*.node-uuid=1a05c9a0-f9cc-4055-8f22-61c6dd314c94
>
>Jun 08 07:25:17 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server...
>Jun 08 07:25:20 dhcp47-76.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server.
>Jun 08 08:53:58 dhcp47-76.lab.eng.blr.redhat.com glusterd[830]: [2018-06-08 08:53:58.338441] C [rpc-clnt.c:465:rpc_clnt_fill_request_info] 0-management: cannot lookup the saved frame corresponding to xid (103)
>[heketi] INFO 2018/06/08 09:53:14 Periodic health check status: node d942fe6c0ee5691b7cc263968f97b650 up=true
>[heketi] INFO 2018/06/08 09:53:14 Cleaned 0 nodes from health cache
>[negroni] Started DELETE /volumes/5db55c483f6984d9917cfe4b3c8b3cbc
>[negroni] Started DELETE /volumes/9c85de66c12db0a72b4d16fe888ff74d
>[negroni] Completed 202 Accepted in 19.305117ms
>[asynchttp] INFO 2018/06/08 09:53:35 asynchttp.go:288: Started job 59c48f7b67bd8d3cccd84a55c1cb7ee8
>[heketi] INFO 2018/06/08 09:53:35 Started async operation: Delete Volume
>[negroni] Started GET /queue/59c48f7b67bd8d3cccd84a55c1cb7ee8
>[negroni] Completed 200 OK in 151.492Âµs
>[negroni] Completed 202 Accepted in 34.138647ms
>[asynchttp] INFO 2018/06/08 09:53:36 asynchttp.go:288: Started job dbbf880df6955eb8f03eb50ffd8e699a
>[heketi] INFO 2018/06/08 09:53:36 Started async operation: Delete Volume
>[negroni] Started GET /queue/dbbf880df6955eb8f03eb50ffd8e699a
>[negroni] Completed 200 OK in 169.592Âµs
>[kubeexec] DEBUG 2018/06/08 09:53:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-122.lab.eng.blr.redhat.com Pod: glusterfs-storage-pg4xc Command: gluster --mode=script snapshot list vol_5db55c483f6984d9917cfe4b3c8b3cbc --xml
>Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
><cliOutput>
>  <opRet>-1</opRet>
>  <opErrno>30806</opErrno>
>  <opErrstr>Volume (vol_5db55c483f6984d9917cfe4b3c8b3cbc) does not exist</opErrstr>
></cliOutput>
>[kubeexec] DEBUG 2018/06/08 09:53:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-76.lab.eng.blr.redhat.com Pod: glusterfs-storage-gxp7c Command: gluster --mode=script snapshot list vol_9c85de66c12db0a72b4d16fe888ff74d --xml
>Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
><cliOutput>
>  <opRet>-1</opRet>
>  <opErrno>30806</opErrno>
>  <opErrstr>Volume (vol_9c85de66c12db0a72b4d16fe888ff74d) does not exist</opErrstr>
></cliOutput>
>[kubeexec] ERROR 2018/06/08 09:53:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume stop vol_5db55c483f6984d9917cfe4b3c8b3cbc force] on glusterfs-storage-pg4xc: Err[command terminated with exit code 1]: Stdout []: Stderr [volume stop: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>]
>[cmdexec] ERROR 2018/06/08 09:53:36 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:144: Unable to stop volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume stop: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[kubeexec] ERROR 2018/06/08 09:53:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume stop vol_9c85de66c12db0a72b4d16fe888ff74d force] on glusterfs-storage-gxp7c: Err[command terminated with exit code 1]: Stdout []: Stderr [volume stop: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>]
>[cmdexec] ERROR 2018/06/08 09:53:36 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:144: Unable to stop volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume stop: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[kubeexec] ERROR 2018/06/08 09:53:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume delete vol_5db55c483f6984d9917cfe4b3c8b3cbc] on glusterfs-storage-pg4xc: Err[command terminated with exit code 1]: Stdout []: Stderr [volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>]
>[cmdexec] ERROR 2018/06/08 09:53:36 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:153: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[heketi] ERROR 2018/06/08 09:53:36 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:525: Unable to delete volume: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[heketi] ERROR 2018/06/08 09:53:36 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:429: Error executing delete volume: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[kubeexec] ERROR 2018/06/08 09:53:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster --mode=script volume delete vol_9c85de66c12db0a72b4d16fe888ff74d] on glusterfs-storage-gxp7c: Err[command terminated with exit code 1]: Stdout []: Stderr [volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>]
>[cmdexec] ERROR 2018/06/08 09:53:36 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:153: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[heketi] ERROR 2018/06/08 09:53:36 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:525: Unable to delete volume: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[heketi] ERROR 2018/06/08 09:53:36 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:429: Error executing delete volume: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[heketi] ERROR 2018/06/08 09:53:36 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:1185: Delete Volume Failed: Unable to delete volume vol_5db55c483f6984d9917cfe4b3c8b3cbc: Unable to execute command on glusterfs-storage-pg4xc: volume delete: vol_5db55c483f6984d9917cfe4b3c8b3cbc: failed: Volume vol_5db55c483f6984d9917cfe4b3c8b3cbc does not exist
>[asynchttp] INFO 2018/06/08 09:53:36 asynchttp.go:292: Completed job 59c48f7b67bd8d3cccd84a55c1cb7ee8 in 796.079381ms
>[heketi] ERROR 2018/06/08 09:53:36 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:1185: Delete Volume Failed: Unable to delete volume vol_9c85de66c12db0a72b4d16fe888ff74d: Unable to execute command on glusterfs-storage-gxp7c: volume delete: vol_9c85de66c12db0a72b4d16fe888ff74d: failed: Volume vol_9c85de66c12db0a72b4d16fe888ff74d does not exist
>[asynchttp] INFO 2018/06/08 09:53:36 asynchttp.go:292: Completed job dbbf880df6955eb8f03eb50ffd8e699a in 793.577264ms
>[negroni] Started GET /queue/59c48f7b67bd8d3cccd84a55c1cb7ee8
>[negroni] Completed 500 Internal Server Error in 154.797Âµs
>[negroni] Started GET /queue/dbbf880df6955eb8f03eb50ffd8e699a
>[negroni] Completed 500 Internal Server Error in 168.215Âµs

Actions: View

Attachments on bug 1590143: 1450325 | 1450328