Bug 1981571

Both the upstream bug and the previous bug here are nova bugs.

from a policy point of view it is never correct to grant a backport to a specific z stream of OSP.

OSP does not provide any sku or add-on that provides EUS(extended update support) and its against our support
policy to deliver hotfixes based on a specific version.

This can be backported to 16.1 for inclusion in the next z stream potentially but it would be released in 16.1.8 not 16.1.2
i believe this will also be included in 16.2.0 which is due some time in the next 6 weeks.

looking at the case the inital description states
Driver failed to attach volume a507dada-3ad0-47d6-a814-2d6a501b31bc at /dev/vdj: libvirt.libvirtError: Requested operation is not valid: target vdj already exists 
the upstream bug was and code path was form detach not attach so its not imidiatly clear that these too are related.

the solution to the upstream detach bug https://review.opendev.org/c/openstack/nova/+/788720 is likely not applicable to the current issue being reported so before
we talk of backports we need to understand why attach is failing.

setting dev_nack repoducer until we can actually triage this issue properly and understand what is failing and how.

if you can provide a detailed bug report of the action they were performing and what vm/volumes failed
so that we can more eaisly review the logs in teh sosrepots we can try and root cause it.

i have not fully read the case but one thing that jumps out at me is the naming schem is vdj

the v implises the hw_disk_bus unset or set to virtio which is our default.
in that mode each volume requires a pci slot in the guest. for that reason when using shift on stack or in any case where you expect to attach more then 1 or 2 volumens we generally recommend that you use hw_disk_bus=scsi hw_scsi_model=virtio-scsi instead.
this will signifcantly increase the number of volumes you can attach.


for shift on stack virtio-scsi is almost always perferable as a result to not hit limiation with avaiable pci slots.

Hi Smooney 

This is what the customer is observing 

- Through OCP a volume is attached to the Openstack instance
- When the volume is detached. 
  - This completes successfully as the disk is no longer present in the OS. 
  - But the disk has not been removed from the libvirt XML.
- When trying to attach a new volume it tries to use the same device I.E. vdj but fails because it is already defined. 

This seems similar to what was described in this bug https://bugzilla.redhat.com/show_bug.cgi?id=1838786

i see so the repuduceer here is a detach that partly succeeds followed by an attach.

i.e. boot a vm attach a volume, detach the volume form the os in the guest but not fully in the xml and then try to attach a new volume resulting in a name collision.
ill loop back with lee. the inital fixes with extending the timeout is really just a tempory hack. if this a result of the detach failure we would likely need to backport the event code
which is somewhat involved and may not be feasible to fix in 16.

(In reply to David Sedgmen from comment #5)
> Hi Smooney 
> 
> This is what the customer is observing 
> 
> - Through OCP a volume is attached to the Openstack instance
> - When the volume is detached. 
>   - This completes successfully as the disk is no longer present in the OS. 
>   - But the disk has not been removed from the libvirt XML.
> - When trying to attach a new volume it tries to use the same device I.E.
> vdj but fails because it is already defined. 
> 
> This seems similar to what was described in this bug
> https://bugzilla.redhat.com/show_bug.cgi?id=1838786

Yeah it sounds the same, does the customer have logs associated with the detach *before* the first attach request that failed?

$ openstack server event list $instance
$ openstack server event show $instance $request-id-of-last-detach-before-attach-failed

^ If they can also provide nova-compute logs referencing that request-id and if possible libvirtd from the same compute we should be able to trace why the device is still present within the libvirt domain.

Obviously to workaround this the customer can manually detach that device from the libvirt domain using virsh, that should allow attach request to succeed again.

IIUC the suggestion was that this bug occurs due to a timeout waiting for the guest OS to detach the volume? From comment 7 I'm also guessing there's no indication of this condition in the API, and therefore it can't be resolved through the API.

Please let me know if there's any way we can robustify shift on stack against this. For example, if the old attachment was in some intermediate or error state we could retry it if that was supported.

(In reply to Matthew Booth from comment #9)
> IIUC the suggestion was that this bug occurs due to a timeout waiting for
> the guest OS to detach the volume? From comment 7 I'm also guessing there's
> no indication of this condition in the API, and therefore it can't be
> resolved through the API.

That was just a suggestion from the initial reporter, I don't think it's the actual cause here as that shouldn't delete the underlying bdm record. As I said before we really need logs from an example detach to determine what happened here to cause the device to remain attached to the domain but leave the bdm deleted in Nova.

> Please let me know if there's any way we can robustify shift on stack
> against this. For example, if the old attachment was in some intermediate or
> error state we could retry it if that was supported.

I can't think of anything other than a hard reboot to workaround this situation for OCP and that seems overkill potentially.

Do you know if OCP is using the os-volume_attachments API to determine when a volume is attached or detached btw? I get the feeling it's firing off the async attach or detach requests and treating the 202's we return as the volume now being attached or detached.

This is the (legacy) in-tree cinder volume driver which is most likely being used here. I haven't checked the CSI driver, but chances are it's the same.

https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/cinder/attacher.go#L160-L166

We're basically doing:

  volume.attach()
  loop:
    volume = get volume()
    if volume.attachments[0].serverID == instance.id:
      break

So https://github.com/kubernetes/kubernetes/blob/5be21c50c269fc1d28e0bd31ab9dcb572ae7fac5/staging/src/k8s.io/legacy-cloud-providers/openstack/openstack_volumes.go#L629 AFAICT? That should really be polling Nova's os-volume_attachments IMHO, I also note none of this code appears to support multi attach?

https://docs.openstack.org/api-ref/compute/?expanded=list-volume-attachments-for-an-instance-detail#list-volume-attachments-for-an-instance

Closing out with INSUFFICIENT_DATA, as requested in c#10 we need logs showing the detach prior to the first attach failure to understand what happened here, without that there's nothing we can do.

(In reply to Lee Yarwood from comment #12)
> So
> https://github.com/kubernetes/kubernetes/blob/
> 5be21c50c269fc1d28e0bd31ab9dcb572ae7fac5/staging/src/k8s.io/legacy-cloud-
> providers/openstack/openstack_volumes.go#L629 AFAICT? That should really be
> polling Nova's os-volume_attachments IMHO, I also note none of this code
> appears to support multi attach?
> 
> https://docs.openstack.org/api-ref/compute/?expanded=list-volume-attachments-
> for-an-instance-detail#list-volume-attachments-for-an-instance

FWIW we had another issue similar to this and I ended up checking the Cinder CSI code. It is the same as this legacy implementation so I've raised the following issue to have it switched over to os-volume_attachments:

[cinder-csi-plugin] Use Nova's os-volume_attachments API to poll attach/detach completion 
https://github.com/kubernetes/cloud-provider-openstack/issues/1645

The needinfo request[s] on this closed bug have been removed as they have been unresolved for 120 days