Created attachment 656026 [details] engine log [Upgrade] [Storage] [Hot Plug] Error message on each plug or unplug disk on NFS storage although the plug and unplug successful. How to reproduce: 1.Create Vm with several disks. 2.Start VM. 3.Plug and unplug disk. Reproduces 100% on several environments on NFS storage. Doesn't happen on FC Storage. Probably won't happen on ISCSI too. 2012-12-02 15:30:54,160 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-69) VM d96ab89d-30f3-4b97-bd09-e6cb97068e0a managed non pluggable device was removed unexpectedly from libvirt: VmDevice {vmId=d96ab89d-30f3-4b97-bd09-e6cb97068e0a, deviceId=66324a57-8dbf-4e80-a26e-edfd7037dd3a, device=disk, type=disk, bootOrder=0, specParams={}, address={bus=0x00, domain=0x0000, type=pci, slot=0x0a, function=0x0}, managed=true, plugged=false, readOnly=false, alias=virtio-disk2} Also the error message format seems to be incorrect, For example "alias=virtio-disk2" is incorrect. There is no alias virtio-disk2. Full log attached.
bug 885984 was cloned from this one for Error message issue. This bug will deal with the VdsUpdateRunTimeInfo ERROR alone. It did reproduce for me, thought not every time. Plug and unplug end with success - ALWAYS. There is some race between plug/unplug and VdsUpdateRunTimeInfo, depends on when exactly VdsUpdateRunTimeInfo actually runs. plug/unplug probably changes data that VdsUpdateRunTimeInfo is trying to retrieve. The error occurs in VdsUpdateRunTimeInfo and has no effect on the success/failure of plug/unplug. It's only detectable in the log. lesik - Why did you mark upgrade in the bug's title?
What basically happens is that on detach of any device (Not necessarily a disk) there can be a race and after the unplug command is sent to VDSM but before the update in the DB, the VdsUpdateRunTimeInfo returns the list of active devices without the unplugged device (Since VDSM already unplugged it) and then VdsUpdateRunTimeInfo handles it as a device which was removed unexpectedly. Given the facts, this is a virt issue so moving to virt team
Same error found on RHEVM 3.2 - SF03 environment ( Hot-Plug and Hot-UnPlug Direct LUN in FC DC environment, with preallocated disk only) RHEVM: rhevm-3.2.0-4.el6ev.noarch VDSM: vdsm-4.10.2-3.0.el6ev.x86_64 LIBVIRT: libvirt-0.10.2-13.el6.x86_64 QEMU & KVM: qemu-kvm-rhev-0.12.1.2-2.348.el6.x86_64 SANLOCK: sanlock-2.6-2.el6.x86_64
Cleared needinfo flag on me since no question was ever asked.
not exactly the same because this one talks about disks and 872165 talks about nics, but it's a general problem for pluggable devices. it seems that the fix for 872165 should solve this case too. *** This bug has been marked as a duplicate of bug 872165 ***