Bug 1506608

Regression, checked in 4.1.7.3 engine & this issue does not occur.

Daniel, weren't you already looking at such an issue?

Daniel, weren't you already looking at such an issue?

Seems that the timeout error is thrown from _waitForDeviceRemoval (vm.py),
which polls device's status using 'device.is_attached_to'.
@Nir - do we check for volume lease on hotunplugDisk? does it require a running guest agent? any special behavior for virtio interface?

(In reply to Daniel Erez from comment #4)
> Seems that the timeout error is thrown from _waitForDeviceRemoval (vm.py),
> which polls device's status using 'device.is_attached_to'.
> @Nir - do we check for volume lease on hotunplugDisk? does it require a
> running guest agent? any special behavior for virtio interface?

I don't know about special behavior, and we did not make any changes in the 
relevant storage code in vdsm. However they were huges changes in virt, so
I suggest to move this to virt.

Francesco, can you take a look at this?

I do not see any error anywhere, the request for removal in libvirt log is
2017-10-26 12:17:57.546+0000: 16740: info : qemuMonitorSend:1034 : QEMU_MONITOR_SEND_MSG: mon=0x7f3ddc8592a0 msg={"execute":"device_del","arguments":{"id":"virtio-disk0"},"id":"libvirt-162"}

which seem to time out eventually. Was the drive actually unplugged or not, do we have guest side logs? Can you get logs from non-gluster storage to rule out a gluster issue?

(In reply to Michal Skrivanek from comment #6)
> I do not see any error anywhere, the request for removal in libvirt log is
> 2017-10-26 12:17:57.546+0000: 16740: info : qemuMonitorSend:1034 :
> QEMU_MONITOR_SEND_MSG: mon=0x7f3ddc8592a0
> msg={"execute":"device_del","arguments":{"id":"virtio-disk0"},"id":"libvirt-
> 162"}
> 
> which seem to time out eventually. Was the drive actually unplugged or not,
> do we have guest side logs? Can you get logs from non-gluster storage to
> rule out a gluster issue?

@Avihai - can you please also check which version of libvirt/qemu is in the 4.1 env that worked.

(In reply to Daniel Erez from comment #7)
> (In reply to Michal Skrivanek from comment #6)
> > I do not see any error anywhere, the request for removal in libvirt log is
> > 2017-10-26 12:17:57.546+0000: 16740: info : qemuMonitorSend:1034 :
> > QEMU_MONITOR_SEND_MSG: mon=0x7f3ddc8592a0
> > msg={"execute":"device_del","arguments":{"id":"virtio-disk0"},"id":"libvirt-
> > 162"}
> > 
> > which seem to time out eventually. Was the drive actually unplugged or not,
> > do we have guest side logs? Can you get logs from non-gluster storage to
> > rule out a gluster issue?
> 
> @Avihai - can you please also check which version of libvirt/qemu is in the
> 4.1 env that worked.

Another thing, did you install an OS on the VM. Hot unplug with virtio is applicable only with guest installed.

(In reply to Daniel Erez from comment #7)
> (In reply to Michal Skrivanek from comment #6)
> > I do not see any error anywhere, the request for removal in libvirt log is
> > 2017-10-26 12:17:57.546+0000: 16740: info : qemuMonitorSend:1034 :
> > QEMU_MONITOR_SEND_MSG: mon=0x7f3ddc8592a0
> > msg={"execute":"device_del","arguments":{"id":"virtio-disk0"},"id":"libvirt-
> > 162"}
> > 
> > which seem to time out eventually. Was the drive actually unplugged or not,
> > do we have guest side logs? Can you get logs from non-gluster storage to
> > rule out a gluster issue?
> 
> @Avihai - can you please also check which version of libvirt/qemu is in the
> 4.1 env that worked.

libvirt - 3.2.0-14.el7_4.3
qemu - 2.9.0-16.el7_4.9

(In reply to Daniel Erez from comment #8)
> (In reply to Daniel Erez from comment #7)
> > (In reply to Michal Skrivanek from comment #6)
> > > I do not see any error anywhere, the request for removal in libvirt log is
> > > 2017-10-26 12:17:57.546+0000: 16740: info : qemuMonitorSend:1034 :
> > > QEMU_MONITOR_SEND_MSG: mon=0x7f3ddc8592a0
> > > msg={"execute":"device_del","arguments":{"id":"virtio-disk0"},"id":"libvirt-
> > > 162"}
> > > 
> > > which seem to time out eventually. Was the drive actually unplugged or not,
> > > do we have guest side logs? Can you get logs from non-gluster storage to
> > > rule out a gluster issue?
> > 
> > @Avihai - can you please also check which version of libvirt/qemu is in the
> > 4.1 env that worked.
> 
> Another thing, did you install an OS on the VM. Hot unplug with virtio is
> applicable only with guest installed.

I rechecked hot-unplug with VIRTIO interface on lastest 4.2:
- VM with OS -> issue does not occur.
- VM without OS -> issue occur.

I also rechecked the same scenario on 4.1 & got the same behavior so this is not a regression as I thought.

Last Q: -about VM without OS scenario:
Hot plug does work but hot unplug does not - Is this known/by design?


Anyway, If this behavior is by design - please close the bug.

(In reply to Avihai from comment #10)
> (In reply to Daniel Erez from comment #8)
> > (In reply to Daniel Erez from comment #7)
> > > (In reply to Michal Skrivanek from comment #6)
> > > > I do not see any error anywhere, the request for removal in libvirt log is
> > > > 2017-10-26 12:17:57.546+0000: 16740: info : qemuMonitorSend:1034 :
> > > > QEMU_MONITOR_SEND_MSG: mon=0x7f3ddc8592a0
> > > > msg={"execute":"device_del","arguments":{"id":"virtio-disk0"},"id":"libvirt-
> > > > 162"}
> > > > 
> > > > which seem to time out eventually. Was the drive actually unplugged or not,
> > > > do we have guest side logs? Can you get logs from non-gluster storage to
> > > > rule out a gluster issue?
> > > 
> > > @Avihai - can you please also check which version of libvirt/qemu is in the
> > > 4.1 env that worked.
> > 
> > Another thing, did you install an OS on the VM. Hot unplug with virtio is
> > applicable only with guest installed.
> 
> I rechecked hot-unplug with VIRTIO interface on lastest 4.2:
> - VM with OS -> issue does not occur.
> - VM without OS -> issue occur.
> 
> I also rechecked the same scenario on 4.1 & got the same behavior so this is
> not a regression as I thought.
> 
> Last Q: -about VM without OS scenario:
> Hot plug does work but hot unplug does not - Is this known/by design?

Indeed, disk hot unplug requires a running OS. Closing..

> 
> 
> Anyway, If this behavior is by design - please close the bug.

I still wonder how did you check it as a regression. Do you have different OSes? Is it automation? Please make sure the test is reproducible in the same conditions!

(In reply to Michal Skrivanek from comment #12)
> I still wonder how did you check it as a regression. Do you have different
> OSes? Is it automation? Please make sure the test is reproducible in the
> same conditions!

I had no idea that we have a known issue/by design with hot-unplug with VIRTIO ONLY when VM bootable disk is without VM so I did not pay attention to how I create the VM .

Anyway It's my bad & I'll pay more attention next time.