Description of problem: ===================== Had a 6 node brick mux enabled cluster, with a 1*3 volume 'ozone' and volume options of 'ozone' set to 'gluster-block'. Was creating 2-3 blocks (ha 1-3) at a time, deleting them with 'unlink-storage' set to 'no', and creating new blocks using the same backend storage file. Did that for about 4-5 rounds and then started creating blocks with size set to different units (was verifying BZ 1545049). That was when creating a block 'ob13' timed out. Below traceback was seen in /var/log/messages. Was not able to take the sosreports and related logs immediately. When I logged in the next day, multiple nodes were down with ABRT messages and I had to force reboot. I am still not able to get node4 and node6 online, but I am assuming that is not going to be needed, as the block-hosting-volume 'ozone' resided on node1, node2 and node3. All logs will be present at: http://rhsqe-repo.lab.eng.blr.redhat.com/sosreports/swetas/<bugnumber> Apr 12 06:38:40 dhcp47-50 gluster-blockd: Traceback (most recent call last): Apr 12 06:38:40 dhcp47-50 gluster-blockd: File "/usr/bin/targetcli", line 121, in <module> Apr 12 06:38:40 dhcp47-50 gluster-blockd: main() Apr 12 06:38:40 dhcp47-50 gluster-blockd: File "/usr/bin/targetcli", line 111, in main Apr 12 06:38:40 dhcp47-50 gluster-blockd: shell.run_interactive() Apr 12 06:38:40 dhcp47-50 gluster-blockd: File "/usr/lib/python2.7/site-packages/configshell_fb/shell.py", line 905, in run_interactive Apr 12 06:38:40 dhcp47-50 gluster-blockd: self._cli_loop() Apr 12 06:38:40 dhcp47-50 gluster-blockd: File "/usr/lib/python2.7/site-packages/configshell_fb/shell.py", line 734, in _cli_loop Apr 12 06:38:40 dhcp47-50 gluster-blockd: self.run_cmdline(cmdline) Apr 12 06:38:40 dhcp47-50 gluster-blockd: File "/usr/lib/python2.7/site-packages/configshell_fb/shell.py", line 848, in run_cmdline Apr 12 06:38:40 dhcp47-50 gluster-blockd: self._execute_command(path, command, pparams, kparams) Apr 12 06:38:40 dhcp47-50 gluster-blockd: File "/usr/lib/python2.7/site-packages/configshell_fb/shell.py", line 823, in _execute_command Apr 12 06:38:40 dhcp47-50 gluster-blockd: result = target.execute_command(command, pparams, kparams) Apr 12 06:38:40 dhcp47-50 gluster-blockd: File "/usr/lib/python2.7/site-packages/configshell_fb/node.py", line 1406, in execute_command Apr 12 06:38:40 dhcp47-50 gluster-blockd: return method(*pparams, **kparams) Apr 12 06:38:40 dhcp47-50 gluster-blockd: File "/usr/lib/python2.7/site-packages/targetcli/ui_backstore.py", line 601, in ui_command_create Apr 12 06:38:40 dhcp47-50 gluster-blockd: so = UserBackedStorageObject(name, size=size, config=config, wwn=wwn) Apr 12 06:38:40 dhcp47-50 gluster-blockd: File "/usr/lib/python2.7/site-packages/rtslib_fb/tcm.py", line 798, in __init__ Apr 12 06:38:40 dhcp47-50 gluster-blockd: self._configure(config, size, wwn) Apr 12 06:38:40 dhcp47-50 gluster-blockd: File "/usr/lib/python2.7/site-packages/rtslib_fb/tcm.py", line 812, in _configure Apr 12 06:38:40 dhcp47-50 gluster-blockd: self._enable() Apr 12 06:38:40 dhcp47-50 gluster-blockd: File "/usr/lib/python2.7/site-packages/rtslib_fb/tcm.py", line 158, in _enable Apr 12 06:38:40 dhcp47-50 gluster-blockd: fwrite(path, "1\n") Apr 12 06:38:40 dhcp47-50 gluster-blockd: File "/usr/lib/python2.7/site-packages/rtslib_fb/utils.py", line 79, in fwrite Apr 12 06:38:40 dhcp47-50 gluster-blockd: file_fd.write(str(string)) Apr 12 06:38:40 dhcp47-50 gluster-blockd: IOError: [Errno 2] No such file or directory Version-Release number of selected component (if applicable): ============================================================= glusterfs-3.12.2-7 gluster-block-0.2.1-17 tcmu-runner-1.2.0-18 kernel-3.10.0-862 How reproducible: ================= 1:1