Bug 1278209

Hi, for this fix we would actually want it to be targeted for 7.0 release. Looks like it has been automatically set to 8.0. Would it be possible for someone to update it, please?

(In reply to openstack-dev from comment #0)
> Description of problem:
> 
> https://bugs.launchpad.net/nova/+bug/1385798
> 
> When disconnecting a volume from an instance the ISCSI multipath connection
> is not always cleaned up correctly. When running the temepest tests we see
> test failures related to this as the connection is not closed, but then it
> is requesting to disconnect through the cinder driver which ends up breaking
> the iscsi connection. The end result being that there are still entries in
> /dev/disk/by-path for the old ISCSI connections, but they are in an error
> state and cannot be used.
> 
> This issue is fixed in the latest os-brick, and a proposed backport of those
> fixes is up for review https://review.openstack.org/#/c/229152

Thanks for the patch but we really can't take it as one huge (+307, -98) change. I propose we use this BZ as a tracker and create individual bugs for issues not already addressed in 7.0.z (we already have the Multipath error parsing patches for example).

Making a start on this now but if you have this patch broken out into individual patches with tests etc then please let me know!

(In reply to openstack-dev from comment #3)
> Hi, for this fix we would actually want it to be targeted for 7.0 release.
> Looks like it has been automatically set to 8.0. Would it be possible for
> someone to update it, please?

Moved to 7.0.z as this isn't required for 8.0 with the move to os-brick.

(In reply to openstack-dev from comment #0)
> Description of problem:
> 
> https://bugs.launchpad.net/nova/+bug/1385798
> 
> When disconnecting a volume from an instance the ISCSI multipath connection
> is not always cleaned up correctly. When running the temepest tests we see
> test failures related to this as the connection is not closed, but then it
> is requesting to disconnect through the cinder driver which ends up breaking
> the iscsi connection. The end result being that there are still entries in
> /dev/disk/by-path for the old ISCSI connections, but they are in an error
> state and cannot be used.
> 
> This issue is fixed in the latest os-brick, and a proposed backport of those
> fixes is up for review https://review.openstack.org/#/c/229152

Can I ask that you provide examples of these failures against the latest OSP 7 builds that we have made available? Several fixes have landed downstream since this bug was created and I want to ensure that there are still issues to be addressed here.

(In reply to Lee Yarwood from comment #5)
> (In reply to openstack-dev from comment #0)
> > Description of problem:
> > 
> > https://bugs.launchpad.net/nova/+bug/1385798
> > 
> > When disconnecting a volume from an instance the ISCSI multipath connection
> > is not always cleaned up correctly. When running the temepest tests we see
> > test failures related to this as the connection is not closed, but then it
> > is requesting to disconnect through the cinder driver which ends up breaking
> > the iscsi connection. The end result being that there are still entries in
> > /dev/disk/by-path for the old ISCSI connections, but they are in an error
> > state and cannot be used.
> > 
> > This issue is fixed in the latest os-brick, and a proposed backport of those
> > fixes is up for review https://review.openstack.org/#/c/229152
> 
> Can I ask that you provide examples of these failures against the latest OSP
> 7 builds that we have made available? Several fixes have landed downstream
> since this bug was created and I want to ensure that there are still issues
> to be addressed here.

We are still seeing customers of ours running into this problem, so I'm assuming the issue still exists. Can you link the fixes that landed that would have addressed this?

(In reply to Patrick East from comment #6)
> We are still seeing customers of ours running into this problem, so I'm
> assuming the issue still exists. Can you link the fixes that landed that
> would have addressed this?

Hello Patrick, 

I've included a list of changes made to nova/virt/libvirt/volume.py in OSP 7 below [1]. IMHO the recent removal of additional rescans during disconnect_volume for iSCSI volumes will likely help here. Again if you can provide actual examples I'd be more than happy to review logs in order to confirm that this should now be addressed.

Also can I ask that you set a NEEDINFO against me when providing any such examples just to ensure I am aware of your update.

Thanks in advance,

Lee

[1] # git log --pretty=email 2015.1.0..HEAD nova/virt/libvirt/volume.py | egrep '(Subject|Resolves)'                                                            
Subject: [PATCH] Use stashed volume connector in _local_cleanup_bdm_volumes
Resolves: rhbz#1351662

Subject: [PATCH] volume: remove iSCSI rescans during disconnect_volume
Resolves: rhbz #1351169

Subject: [PATCH] Optimize multipath call to identify IQN
Resolves: rhbz#1331256

Subject: [PATCH] libvirt: Reduce iscsiadm use when using multipath
Resolves: rhbz#1334161

Subject: [PATCH] libvirt: Remove devices from the connection_info data dict
Resolves: rhbz #1313624

Subject: [PATCH] volume: Fix the removal of multipath backed iSCSI LUNs
Resolves: rhbz#1298283

Subject: [PATCH] libvirt: Detect and remove multipath devices on disconnect
Resolves rhbz: 1273473

Subject: [PATCH] libvirt: Retry multipathd queries during FC volume connection
Resolves rhbz: 1273473

Subject: [PATCH] libvirt: Parse FCoE sysfs device paths
Resolves rhbz: 1274054

Subject: [PATCH] Multipath commands with error messages in stdout fail to
Resolves: rhbz#1275937

Subject: [PATCH] libvirt: Revert _get_host_devices return value
Resolves: rhbz#1268051

Subject: [PATCH] libvirt: Enhance iSCSI volume multipath support
Resolves: rhbz#1228295

Subject: [PATCH] Handle FC LUN IDs greater 255 correctly on s390x

(In reply to Lee Yarwood from comment #7)
> (In reply to Patrick East from comment #6)
> > We are still seeing customers of ours running into this problem, so I'm
> > assuming the issue still exists. Can you link the fixes that landed that
> > would have addressed this?
> 
> Hello Patrick, 
> 
> I've included a list of changes made to nova/virt/libvirt/volume.py in OSP 7
> below [1]. IMHO the recent removal of additional rescans during
> disconnect_volume for iSCSI volumes will likely help here. Again if you can
> provide actual examples I'd be more than happy to review logs in order to
> confirm that this should now be addressed.
> 
> Also can I ask that you set a NEEDINFO against me when providing any such
> examples just to ensure I am aware of your update.
> 
> Thanks in advance,
> 
> Lee
> 
> [1] # git log --pretty=email 2015.1.0..HEAD nova/virt/libvirt/volume.py |
> egrep '(Subject|Resolves)'                                                  
> 
> Subject: [PATCH] Use stashed volume connector in _local_cleanup_bdm_volumes
> Resolves: rhbz#1351662
> 
> Subject: [PATCH] volume: remove iSCSI rescans during disconnect_volume
> Resolves: rhbz #1351169
> 
> Subject: [PATCH] Optimize multipath call to identify IQN
> Resolves: rhbz#1331256
> 
> Subject: [PATCH] libvirt: Reduce iscsiadm use when using multipath
> Resolves: rhbz#1334161
> 
> Subject: [PATCH] libvirt: Remove devices from the connection_info data dict
> Resolves: rhbz #1313624
> 
> Subject: [PATCH] volume: Fix the removal of multipath backed iSCSI LUNs
> Resolves: rhbz#1298283
> 
> Subject: [PATCH] libvirt: Detect and remove multipath devices on disconnect
> Resolves rhbz: 1273473
> 
> Subject: [PATCH] libvirt: Retry multipathd queries during FC volume
> connection
> Resolves rhbz: 1273473
> 
> Subject: [PATCH] libvirt: Parse FCoE sysfs device paths
> Resolves rhbz: 1274054
> 
> Subject: [PATCH] Multipath commands with error messages in stdout fail to
> Resolves: rhbz#1275937
> 
> Subject: [PATCH] libvirt: Revert _get_host_devices return value
> Resolves: rhbz#1268051
> 
> Subject: [PATCH] libvirt: Enhance iSCSI volume multipath support
> Resolves: rhbz#1228295
> 
> Subject: [PATCH] Handle FC LUN IDs greater 255 correctly on s390x

Thanks Lee!

Closing this out as a duplicate of RHBZ#1410046 where we recently fixed a long standing issue removing path devices when using backends that only provided single target_{portal,iqn,lun} details via Cinder.

*** This bug has been marked as a duplicate of bug 1410046 ***

Dup -- QE will decide about automating the original