Bug 1566828 - Block create times out, with a targetcli crash
Summary: Block create times out, with a targetcli crash
Keywords:
Status: CLOSED WORKSFORME
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: tcmu-runner
Version: cns-3.9
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: ---
Assignee: Prasanna Kumar Kalever
QA Contact: Rahul Hinduja
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-04-13 05:30 UTC by Sweta Anandpara
Modified: 2018-09-18 08:02 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-06-25 08:27:12 UTC
Target Upstream Version:


Attachments (Terms of Use)

Description Sweta Anandpara 2018-04-13 05:30:31 UTC
Description of problem:
=====================
Had a 6 node brick mux enabled cluster, with a 1*3 volume 'ozone' and volume options of 'ozone' set to 'gluster-block'. 

Was creating 2-3 blocks (ha 1-3) at a time, deleting them with 'unlink-storage' set to 'no', and creating new blocks using the same backend storage file. Did that for about 4-5 rounds and then started creating blocks with size set to different units (was verifying BZ 1545049). That was when creating a block 'ob13' timed out. Below traceback was seen in /var/log/messages.

Was not able to take the sosreports and related logs immediately. When I logged in the next day, multiple nodes were down with ABRT messages and I had to force reboot. I am still not able to get node4 and node6 online, but I am assuming that is not going to be needed, as the block-hosting-volume 'ozone' resided on node1, node2 and node3. 

All logs will be present at:  http://rhsqe-repo.lab.eng.blr.redhat.com/sosreports/swetas/<bugnumber>

Apr 12 06:38:40 dhcp47-50 gluster-blockd: Traceback (most recent call last):
Apr 12 06:38:40 dhcp47-50 gluster-blockd: File "/usr/bin/targetcli", line 121, in <module>
Apr 12 06:38:40 dhcp47-50 gluster-blockd: main()
Apr 12 06:38:40 dhcp47-50 gluster-blockd: File "/usr/bin/targetcli", line 111, in main
Apr 12 06:38:40 dhcp47-50 gluster-blockd: shell.run_interactive()
Apr 12 06:38:40 dhcp47-50 gluster-blockd: File "/usr/lib/python2.7/site-packages/configshell_fb/shell.py", line 905, in run_interactive
Apr 12 06:38:40 dhcp47-50 gluster-blockd: self._cli_loop()
Apr 12 06:38:40 dhcp47-50 gluster-blockd: File "/usr/lib/python2.7/site-packages/configshell_fb/shell.py", line 734, in _cli_loop
Apr 12 06:38:40 dhcp47-50 gluster-blockd: self.run_cmdline(cmdline)
Apr 12 06:38:40 dhcp47-50 gluster-blockd: File "/usr/lib/python2.7/site-packages/configshell_fb/shell.py", line 848, in run_cmdline
Apr 12 06:38:40 dhcp47-50 gluster-blockd: self._execute_command(path, command, pparams, kparams)
Apr 12 06:38:40 dhcp47-50 gluster-blockd: File "/usr/lib/python2.7/site-packages/configshell_fb/shell.py", line 823, in _execute_command
Apr 12 06:38:40 dhcp47-50 gluster-blockd: result = target.execute_command(command, pparams, kparams)
Apr 12 06:38:40 dhcp47-50 gluster-blockd: File "/usr/lib/python2.7/site-packages/configshell_fb/node.py", line 1406, in execute_command
Apr 12 06:38:40 dhcp47-50 gluster-blockd: return method(*pparams, **kparams)
Apr 12 06:38:40 dhcp47-50 gluster-blockd: File "/usr/lib/python2.7/site-packages/targetcli/ui_backstore.py", line 601, in ui_command_create
Apr 12 06:38:40 dhcp47-50 gluster-blockd: so = UserBackedStorageObject(name, size=size, config=config, wwn=wwn)
Apr 12 06:38:40 dhcp47-50 gluster-blockd: File "/usr/lib/python2.7/site-packages/rtslib_fb/tcm.py", line 798, in __init__
Apr 12 06:38:40 dhcp47-50 gluster-blockd: self._configure(config, size, wwn)
Apr 12 06:38:40 dhcp47-50 gluster-blockd: File "/usr/lib/python2.7/site-packages/rtslib_fb/tcm.py", line 812, in _configure
Apr 12 06:38:40 dhcp47-50 gluster-blockd: self._enable()
Apr 12 06:38:40 dhcp47-50 gluster-blockd: File "/usr/lib/python2.7/site-packages/rtslib_fb/tcm.py", line 158, in _enable
Apr 12 06:38:40 dhcp47-50 gluster-blockd: fwrite(path, "1\n")
Apr 12 06:38:40 dhcp47-50 gluster-blockd: File "/usr/lib/python2.7/site-packages/rtslib_fb/utils.py", line 79, in fwrite
Apr 12 06:38:40 dhcp47-50 gluster-blockd: file_fd.write(str(string))
Apr 12 06:38:40 dhcp47-50 gluster-blockd: IOError: [Errno 2] No such file or directory


Version-Release number of selected component (if applicable):
=============================================================
glusterfs-3.12.2-7
gluster-block-0.2.1-17
tcmu-runner-1.2.0-18
kernel-3.10.0-862

How reproducible:
=================
1:1


Note You need to log in before you can comment on or make changes to this bug.