Bug 1589749
| Summary: | Hot-plugged disk deactivation failed after attaching it to a VM with VIRTIO interface | ||||||
|---|---|---|---|---|---|---|---|
| Product: | [oVirt] ovirt-engine | Reporter: | Eyal Shenitzky <eshenitz> | ||||
| Component: | BLL.Storage | Assignee: | Ahmad Khiet <akhiet> | ||||
| Status: | CLOSED NOTABUG | QA Contact: | Evelina Shames <eshames> | ||||
| Severity: | medium | Docs Contact: | |||||
| Priority: | unspecified | ||||||
| Version: | 4.3.0 | CC: | aefrat, bugs, eshames, kwolf, lsvaty, nsoffer, rbarry, tnisan | ||||
| Target Milestone: | ovirt-4.3.5 | Flags: | pm-rhel:
ovirt-4.3+
|
||||
| Target Release: | --- | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2019-06-02 12:17:46 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | Storage | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Bug Depends On: | |||||||
| Bug Blocks: | 1583046 | ||||||
| Attachments: |
|
||||||
Hotplugged Disk deactivation will fail when disk is attached with 'virtio' interface. Steps to reproduce: 1. Create a VM with a disk 2. Run the VM 3. Attach an existing (floating) disk to the VM with 'virtio' interface 4. Try to deactivate the disk Reproduce also on data-center with 4.1 compatibility level Still reproducible? Avihai, can you do us the honor? :) Evelina, can you please check it out? Still reproducible on 4.3.3.3-0.1
the error is in vdsm :
2019-05-22 07:14:45,495-0400 INFO (jsonrpc/1) [api.virt] FINISH hotunplugDisk return={'status': {'message': 'Timeout detaching <Drive name=vda, type=file, path=/rhev/data-center/mnt/10.10.10.10:_nfs_sd1/d28a5053-57a5-4c6b-9e16-cfe1b0edb32d/images/85971925-d12d-4e1e-bc69-09a1d20f7cac/5651f128-9edf-46be-8021-f011ca7c0d9e threshold=unset at 0x7f4af2783bc0>', 'code': 46}} from=::ffff:10.35.1.173,60340, flow_id=24fe9444-56c2-4539-8e0a-2208d79c4b3c, vmId=a9cf8941-9c19-4452-8acf-90e2a83b3011 (api:54)
Following the bug description, I can reproduce every time. the bug reproduces only with 'virtio' interface every time it fails. but if the disk with 'virtio-ISCSI' interface, unplugging the disk works successfully. the storage has no filesystem.it's empty. only created and attached. Created attachment 1574358 [details]
logs form qemu, vm and vdsm
@kwolf maybe its a bug in qemu? Probably the guest OS is not releasing the PCI device. Can you attach relevant guest logs? logs from /var/log/messages when performing the unplugging for the disk:
May 30 08:51:18 localhost systemd: Started Virtual Machine qemu-7-v1.
May 30 08:51:18 localhost systemd-machined: New machine qemu-7-v1.
May 30 08:51:18 localhost kvm: 2 guests now active
May 30 08:51:19 localhost kernel: connection1:0: detected conn error (1020)
May 30 08:51:19 localhost iscsid: conn 0 login rejected: initiator error - target not found (02/03)
May 30 08:51:19 localhost iscsid: Kernel reported iSCSI connection 1:0 error (1020 - ISCSI_ERR_TCP_CONN_CLOSE: TCP connection closed) state (1)
May 30 08:51:21 localhost libvirtd: 2019-05-30 12:51:21.795+0000: 11353: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU guest agent is not connected
May 30 08:51:21 localhost vdsm[31911]: ERROR Shutdown by QEMU Guest Agent failed#012Traceback (most recent call last):#012 File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 5289, in qemuGuestAgentShutdown#012 self._dom.shu
tdownFlags(libvirt.VIR_DOMAIN_SHUTDOWN_GUEST_AGENT)#012 File "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py", line 100, in f#012 ret = attr(*args, **kwargs)#012 File "/usr/lib/python2.7/site-packages/vdsm/common/libvirtconn
ection.py", line 131, in wrapper#012 ret = f(*args, **kwargs)#012 File "/usr/lib/python2.7/site-packages/vdsm/common/function.py", line 94, in wrapper#012 return func(inst, *args, **kwargs)#012 File "/usr/lib64/python2.7/site-pack
ages/libvirt.py", line 2517, in shutdownFlags#012 if ret == -1: raise libvirtError ('virDomainShutdownFlags() failed', dom=self)#012libvirtError: Guest agent is not responding: QEMU guest agent is not connected
May 30 08:51:21 localhost kernel: connection1:0: detected conn error (1020)
BTW, the VM with no OS PCI hotplug requires OS cooperation, so without an OS, the device will never be released. This is how the hardware works and therefore expected. Maybe the user interface could make clearer why this fails, but otherwise I suggest closing this as NOTABUG. Closing based on comment #14 |
Description of problem: When having a running VM, After attaching a disk to it via REST-API using the following command: POST ovirt-engine/api/vms/123/diskattachments <disk_attachment> <disk id='123'/> <interface>virtio</interface> <bootable>false</bootable> <active>true</active> </disk_attachment> Disk deactivation failed with the following error: 2018-06-11 13:10:17,487+03 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HotUnPlugDiskVDSCommand] (EE-ManagedThreadFactory-engine-Thread-2734) [7b8aa5cc-a02e-4aa0-813b-abd777a8f3ca] Failed in 'HotUnPlugDiskVDS' method 2018-06-11 13:10:17,500+03 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-2734) [7b8aa5cc-a02e-4aa0-813b-abd777a8f3ca] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM h1 command HotUnPlugDiskVDS failed: Timeout detaching <Drive name=vda, type=block, path=/rhev/data-center/mnt/blockSD/9b5fe20f-0ec3-41b3-b20c-a2eeab444297/images/dc4978f4-f978-4016-80bf-6cfb16268c1d/db1dc469-ba6c-4a5b-8172-e3cea0c7a722 threshold=unset at 0x1400e90> 2018-06-11 13:10:17,501+03 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HotUnPlugDiskVDSCommand] (EE-ManagedThreadFactory-engine-Thread-2734) [7b8aa5cc-a02e-4aa0-813b-abd777a8f3ca] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HotUnPlugDiskVDSCommand' return value 'StatusOnlyReturn [status=Status [code=46, message=Timeout detaching <Drive name=vda, type=block, path=/rhev/data-center/mnt/blockSD/9b5fe20f-0ec3-41b3-b20c-a2eeab444297/images/dc4978f4-f978-4016-80bf-6cfb16268c1d/db1dc469-ba6c-4a5b-8172-e3cea0c7a722 threshold=unset at 0x1400e90>]]' 2018-06-11 13:10:17,501+03 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HotUnPlugDiskVDSCommand] (EE-ManagedThreadFactory-engine-Thread-2734) [7b8aa5cc-a02e-4aa0-813b-abd777a8f3ca] HostName = h1 2018-06-11 13:10:17,501+03 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HotUnPlugDiskVDSCommand] (EE-ManagedThreadFactory-engine-Thread-2734) [7b8aa5cc-a02e-4aa0-813b-abd777a8f3ca] Command 'HotUnPlugDiskVDSCommand(HostName = h1, HotPlugDiskVDSParameters:{hostId='8edde004-8086-4f01-b1d8-dc31e25f67d6', vmId='e437ce8e-2231-4c1d-b710-da49369c450c', diskId='dc4978f4-f978-4016-80bf-6cfb16268c1d', addressMap='[bus=0x00, domain=0x0000, function=0x0, slot=0x07, type=pci]'})' execution failed: VDSGenericException: VDSErrorException: Failed to HotUnPlugDiskVDS, error = Timeout detaching <Drive name=vda, type=block, path=/rhev/data-center/mnt/blockSD/9b5fe20f-0ec3-41b3-b20c-a2eeab444297/images/dc4978f4-f978-4016-80bf-6cfb16268c1d/db1dc469-ba6c-4a5b-8172-e3cea0c7a722 threshold=unset at 0x1400e90>, code = 46 2018-06-11 13:10:17,501+03 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HotUnPlugDiskVDSCommand] (EE-ManagedThreadFactory-engine-Thread-2734) [7b8aa5cc-a02e-4aa0-813b-abd777a8f3ca] FINISH, HotUnPlugDiskVDSCommand, return: , log id: e9d1778 2018-06-11 13:10:17,501+03 ERROR [org.ovirt.engine.core.bll.storage.disk.HotUnPlugDiskFromVmCommand] (EE-ManagedThreadFactory-engine-Thread-2734) [7b8aa5cc-a02e-4aa0-813b-abd777a8f3ca] Command 'org.ovirt.engine.core.bll.storage.disk.HotUnPlugDiskFromVmCommand' failed: EngineException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to HotUnPlugDiskVDS, error = Timeout detaching <Drive name=vda, type=block, path=/rhev/data-center/mnt/blockSD/9b5fe20f-0ec3-41b3-b20c-a2eeab444297/images/dc4978f4-f978-4016-80bf-6cfb16268c1d/db1dc469-ba6c-4a5b-8172-e3cea0c7a722 threshold=unset at 0x1400e90>, code = 46 (Failed with error FailedToUnPlugDisk and code 46) 2018-06-11 13:10:17,509+03 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-2734) [7b8aa5cc-a02e-4aa0-813b-abd777a8f3ca] EVENT_ID: USER_FAILED_HOTUNPLUG_DISK(2,003), Failed to unplug disk ttt from VM test (User: admin@internal-authz). Version-Release number of selected component (if applicable): master 4.3.0 - from commit 817759dfe6a2e04c4b67f9036069d3cec6cac6cf How reproducible: 100% Steps to Reproduce: 1. Create a VM with a disk 2. Run the VM 3. Attach an existing (floating) disk to the VM via REST-API 4. Try to deactivate the disk Actual results: Disk deactivation failed Expected results: Disk deactivation should succeed Additional info: there is a difference between the driver XML when hot plugging a disk via the UI and the REST-API, a line is missing in the driver while attaching the disk via REST: <address bus="0" controller="0" unit="2" type="drive" target="0"/> ---------------------------UI-------------------------------------------- ml version="1.0" encoding="UTF-8"?><hotplug> <devices> <disk snapshot="no" type="block" device="disk"> <target dev="sda" bus="scsi"/> <source dev="/rhev/data-center/mnt/blockSD/9b5fe20f-0ec3-41b3-b20c-a2eeab444297/images/a8f855cb-9b4f-450d-b751-ca7e1de27e08/515efc41-ff89-4177-b5c0-00bf1395b945"/> <driver name="qemu" io="native" type="raw" error_policy="stop" cache="none"/> <alias name="ua-a8f855cb-9b4f-450d-b751-ca7e1de27e08"/> <address bus="0" controller="0" unit="2" type="drive" target="0"/> <serial>a8f855cb-9b4f-450d-b751-ca7e1de27e08</serial> </disk> </devices> <metadata xmlns:ovirt-vm="http://ovirt.org/vm/1.0"> <ovirt-vm:vm> <ovirt-vm:device devtype="disk" name="sda"> <ovirt-vm:poolID>6c477009-de73-4d0c-99f2-3b2da3eedb08</ovirt-vm:poolID> <ovirt-vm:volumeID>515efc41-ff89-4177-b5c0-00bf1395b945</ovirt-vm:volumeID> <ovirt-vm:imageID>a8f855cb-9b4f-450d-b751-ca7e1de27e08</ovirt-vm:imageID> <ovirt-vm:domainID>9b5fe20f-0ec3-41b3-b20c-a2eeab444297</ovirt-vm:domainID> </ovirt-vm:device> </ovirt-vm:vm> </metadata> </hotplug> ---------------------REST-------------------------------------- xml version="1.0" encoding="UTF-8"?><hotplug> <devices> <disk snapshot="no" type="block" device="disk"> <target dev="vda" bus="virtio"/> <source dev="/rhev/data-center/mnt/blockSD/9b5fe20f-0ec3-41b3-b20c-a2eeab444297/images/a8f855cb-9b4f-450d-b751-ca7e1de27e08/515efc41-ff89-4177-b5c0-00bf1395b945"/> <driver name="qemu" io="native" type="raw" error_policy="stop" cache="none"/> <alias name="ua-a8f855cb-9b4f-450d-b751-ca7e1de27e08"/> <serial>a8f855cb-9b4f-450d-b751-ca7e1de27e08</serial> </disk> </devices> <metadata xmlns:ovirt-vm="http://ovirt.org/vm/1.0"> <ovirt-vm:vm> <ovirt-vm:device devtype="disk" name="vda"> <ovirt-vm:poolID>6c477009-de73-4d0c-99f2-3b2da3eedb08</ovirt-vm:poolID> <ovirt-vm:volumeID>515efc41-ff89-4177-b5c0-00bf1395b945</ovirt-vm:volumeID> <ovirt-vm:imageID>a8f855cb-9b4f-450d-b751-ca7e1de27e08</ovirt-vm:imageID> <ovirt-vm:domainID>9b5fe20f-0ec3-41b3-b20c-a2eeab444297</ovirt-vm:domainID> </ovirt-vm:device> </ovirt-vm:vm> </metadata> </hotplug>