Bug 1659195

Summary: targetcli 'delete' option fails with stack trace on iscsi block device (Previously incorrectly marked as NOTABUG)
Product: Red Hat Enterprise Linux 7 Reporter: Alex Bensch <alexander.bensch>
Component: targetcliAssignee: Maurizio Lombardi <mlombard>
Status: CLOSED CURRENTRELEASE QA Contact: Martin Hoyer <mhoyer>
Severity: high Docs Contact:
Priority: unspecified    
Version: 7.5CC: alexander.bensch, mchristi, mhoyer
Target Milestone: rc   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-03-20 15:30:12 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Alex Bensch 2018-12-13 19:26:27 UTC
Description of problem:
While configuring a two-node DRBD HA cluster using PCS, I ran into a failure where an iSCSILogicalUnit resource failed to migrate. I traced the issue to targetcli, and through some experimentation I found that the 'delete' command caused a failure in the process. A solution was also identified.

Version-Release number of selected component (if applicable):
targetcli shell version 2.1.fb46

How reproducible:
Follow same steps as detailed in Red Hat Bugzilla bug 1596167.

Steps to Reproduce:
1. Create a /backstores/fileio/ object
#dd if=/dev/zero of=/tmp/backstorage count=2000 bs=1024K
2000+0 records in
2000+0 records out
2097152000 bytes (2.1 GB) copied, 0.606228 s, 3.5 GB/s

# targetcli /backstores/fileio/ create device.storage_pool /tmp/backstorage
Created fileio device.storage_pool with size 2097152000

2. try to delete it
# targetcli /backstores/fileio/ delete device.storage_pool
delete() got an unexpected keyword argument 'save'

3.delete it in interactive console

Actual results:
# targetcli
targetcli shell version 2.1.fb46
Copyright 2011-2013 by Datera, Inc and others.
For help on commands, type 'help'.

/> ls
o- / ......................................................................................................................... [...]
  o- backstores .............................................................................................................. [...]
  | o- block .................................................................................................. [Storage Objects: 0]
  | o- fileio ................................................................................................. [Storage Objects: 1]
  | | o- device.storage_pool .................................................... [/tmp/backstorage (2.0GiB) write-back deactivated]
  | |   o- alua ................................................................................................... [ALUA Groups: 1]
  | |     o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized]
  | o- pscsi .................................................................................................. [Storage Objects: 0]
  | o- ramdisk ................................................................................................ [Storage Objects: 0]
  o- iscsi ............................................................................................................ [Targets: 0]
  o- loopback ......................................................................................................... [Targets: 0]
  o- xen_pvscsi ....................................................................................................... [Targets: 0]
/> backstores/fileio/ delete device.storage_pool 
Traceback (most recent call last):
  File "/usr/bin/targetcli", line 122, in <module>
    main()
  File "/usr/bin/targetcli", line 112, in main
    shell.run_interactive()
  File "/usr/lib/python2.7/site-packages/configshell_fb/shell.py", line 905, in run_interactive
    self._cli_loop()
  File "/usr/lib/python2.7/site-packages/configshell_fb/shell.py", line 734, in _cli_loop
    self.run_cmdline(cmdline)
  File "/usr/lib/python2.7/site-packages/configshell_fb/shell.py", line 848, in run_cmdline
    self._execute_command(path, command, pparams, kparams)
  File "/usr/lib/python2.7/site-packages/configshell_fb/shell.py", line 823, in _execute_command
    result = target.execute_command(command, pparams, kparams)
  File "/usr/lib/python2.7/site-packages/configshell_fb/node.py", line 1406, in execute_command
    return method(*pparams, **kparams)
  File "/usr/lib/python2.7/site-packages/targetcli/ui_backstore.py", line 296, in ui_command_delete
    child.rtsnode.delete(save=save)
TypeError: delete() got an unexpected keyword argument 'save'

Expected results:
Device is successfully deleted.

Additional info:
This issue was solved on my system by changing line 296 in /usr/lib/python2.7/site-packages/targetcli/ui_backstore.py from 

child.rtsnode.delete(save=save)

to

child.rtsnode.delete()

For some reason, the aforementioned Bugzilla bug with near-identical symptoms was marked as CLOSED NOTABUG despite clearly described unintended behavior. 

This issue also breaks PCS HA functionality for iSCSI resources.

Comment 2 Maurizio Lombardi 2018-12-14 10:17:28 UTC
Hello,

(In reply to Alex Bensch from comment #0)
> Description of problem:
> While configuring a two-node DRBD HA cluster using PCS, I ran into a failure
> where an iSCSILogicalUnit resource failed to migrate. I traced the issue to
> targetcli, and through some experimentation I found that the 'delete'
> command caused a failure in the process. A solution was also identified.
> 
> Version-Release number of selected component (if applicable):
> targetcli shell version 2.1.fb46

Are you running rhel7.5, correct?
Can you write here the output of the following command?

# rpm -qa | egrep "targetcli|rtslib"

Comment 3 Martin Hoyer 2018-12-14 10:19:24 UTC
I was unable to reproduce this with RHEL-7.6.
Please make sure you have following package versions:
targetcli-2.1.fb46-7.el7.noarch
python-configshell-1.1.fb23-5.el7.noarch
python-rtslib-2.1.fb63-13.el7.noarch

Comment 4 Martin Hoyer 2018-12-14 10:29:42 UTC
For 7.5 z-stream it would be:
targetcli-2.1.fb46-6.el7_5.noarch
python-rtslib-2.1.fb63-12.el7_5.noarch

It is reproducible if you update targetcli package only to z-stream version.

Comment 5 Alex Bensch 2018-12-14 17:53:03 UTC
(In reply to Maurizio Lombardi from comment #2)
> Hello,
> 
> (In reply to Alex Bensch from comment #0)
> > Description of problem:
> > While configuring a two-node DRBD HA cluster using PCS, I ran into a failure
> > where an iSCSILogicalUnit resource failed to migrate. I traced the issue to
> > targetcli, and through some experimentation I found that the 'delete'
> > command caused a failure in the process. A solution was also identified.
> > 
> > Version-Release number of selected component (if applicable):
> > targetcli shell version 2.1.fb46
> 
> Are you running rhel7.5, correct?
> Can you write here the output of the following command?
> 
> # rpm -qa | egrep "targetcli|rtslib"

Output is:

targetcli-2.1.fb46-6.el7_5.noarch
python-rtslib-2.1.fb63-5.el7.noarch

And yes, RHEL 7.5. Specifically "Red Hat Enterprise Linux Workstation release 7.5 (Maipo)" according to `cat /etc/*-release`, if that helps at all. Not super familiar with the RHEL environment, unfortunately.

Comment 6 Maurizio Lombardi 2018-12-17 09:23:24 UTC
(In reply to Alex Bensch from comment #5)
> Output is:
> 
> targetcli-2.1.fb46-6.el7_5.noarch
> python-rtslib-2.1.fb63-5.el7.noarch
> 
> And yes, RHEL 7.5. Specifically "Red Hat Enterprise Linux Workstation
> release 7.5 (Maipo)" according to `cat /etc/*-release`, if that helps at
> all. Not super familiar with the RHEL environment, unfortunately.

Ok, it should be sufficient to run this:

#yum update targetcli python-rtslib python-configshell
to get rid of the issue.
This is a known problem with dependencies and will be fixed soon.

Comment 7 Maurizio Lombardi 2019-03-20 15:30:12 UTC
Should be fixed in the current release.