Hide Forgot
1. Proposed title of this feature request Provide alternatives to avoid guest hang due to unreachable isodomain . 3. What is the nature and description of the request? Guests with iso images attached go into hang state (Seen as 'Not Responding' in portal) , if the iso domain is unreachable & the guest tries to access any data from the cdrom . In my reproducer (rhel6 guest), the console remained 'hung' until the iso domain was reproducible . Reproduced using iptables on NFS server providing iso domain with DROP & REJECT . Need option to change how the qemu-kvm replies back to the guest, such that the guest does not hang . 4. Why does the customer need this? (List the business requirements here) The iso images are attached to large number of guests . If at all the iso domain goes down , all guests are at risk for being in hung state . 5. How would the customer like to achieve this? (List the functional requirements here) Add option via portal in RunOnce & Change CD through which , if the isodomain is inaccessible , the error is immediately passed to guest in such a way that no hang is noticed on guest . 6. For each functional requirement listed, specify how Red Hat and the customer can test to confirm the requirement is successfully implemented. 7. Is there already an existing RFE upstream or in Red Hat Bugzilla? No. But a similar RFE is raised for error handling for disks . https://bugzilla.redhat.com/show_bug.cgi?id=1024428 8. Does the customer have any specific timeline dependencies and which release would they like to target (i.e. RHEL5, RHEL6)? No 9. Is the sales team involved in this request and do they have any additional input? N 10. List any affected packages or components. qemu-kvm-rhev , vdsm 11. Would the customer be able to assist in testing this functionality if implemented? yes Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
Allon, could it be done with different ioerror handling per disk?
(In reply to Michal Skrivanek from comment #2) > Allon, could it be done with different ioerror handling per disk? I believe it could, but afaik, Kevin Wolf from qemu is adamantly against it (see, e.g., bug 1024428). Kevin - can you provide some insight to this please?
(In reply to Allon Mureinik from comment #3) > I believe it could, but afaik, Kevin Wolf from qemu is adamantly against it > (see, e.g., bug 1024428). > Kevin - can you provide some insight to this please? I am not against using other error options, after all there's a reason why they are options. We just need to check that changing it is the correct solution for the problem at hand; and for the customer problem in bug 1024428 it was clearly not the right solution (it hid the symptoms of a qemu bug for some cases, but it caused new problems in other cases). This BZ might be one of the cases that justify using rerror=report. In contrast to the other BZ, you are really talking about host-side errors here. Read (and even worse write) errors usually make the OS think that a disk has died and refuse to keep using it, so in many cases it would not be useful behaviour. Here we're talking about CD-ROM, i.e. removable media, so ejecting and reloading the image should restore a working state after having reported an error to the guest. So I would say reporting errors to the guest is fine in this specific case. Having said all of that, are you sure this would be a complete fix? Not sure what the 'Not Responding' state in RHEV really means, but qemu can't report anything to the guest until the kernel has failed the request, and it may appear to be completely hanging in some cases before this happened. So you need to make sure that NFS is configured correctly to return an error after a short timeout.
Not Responding reflects the fact that the libvirt process is hung in querying the qemu which is likely in D state Not much can be done about that on NFS, so one alternative is to use a different storage type where errors are reported reliably and immediately. I think indeed the problem as it is stated doesn't need any changes in error reporting or handling, it won't help on NFS
Scott, could you please review? I'd suggest closing
Any downside in reporting the error to the guest if the disk is the cdrom?
(In reply to Yaniv Kaul from comment #15) > Any downside in reporting the error to the guest if the disk is the cdrom? Not that I can imagine.
(In reply to Yaniv Kaul from comment #15) > Any downside in reporting the error to the guest if the disk is the cdrom? The downside is what I mentioned in comment #6, it doesn't solve anything on NFS unless you change the NFS retries/timeouts to something reasonable. If NFS is not a concern then it can be done today via REST API propagate_errors parameter
please test - no change in behavior for default NFS parameters - errors are reported in guest for short timeouts (seconds, perhaps)
*** Bug 1497173 has been marked as a duplicate of this bug. ***
Should this be modified? I see your patch is merged.
(In reply to Yaniv Lavi (Dary) from comment #33) > Should this be modified? I see your patch is merged. Not yet, it didn't work because of the recent 'engine xml' changes in VDSM. I see that https://gerrit.ovirt.org/#/c/81086/ was abandoned - Francesco, so is it supposed to work now or do we wait for the initialization of the storage devices from the xml to get in?
(In reply to Arik from comment #34) > (In reply to Yaniv Lavi (Dary) from comment #33) > > Should this be modified? I see your patch is merged. > > Not yet, it didn't work because of the recent 'engine xml' changes in VDSM. > I see that https://gerrit.ovirt.org/#/c/81086/ was abandoned - Francesco, so > is it supposed to work now or do we wait for the initialization of the > storage devices from the xml to get in? The plan is: 1. merge the remaining patches to enable storage devices to be initialized from XML - this is needed anyway and will solve this issue for 4.2 Those patches are mostly fixes or minor things, so we should get them in at good pace. 2. resume 81086 if needed for non-Engine-XML flows (e.g. backport?)
Created attachment 1382398 [details] engine
Created attachment 1382411 [details] engine & vdsm logs
The bug was re-assigned because of the following tests results. Test Steps: 1. Run VM with CD attached from ISO domain. in <vdsm-client VM getStats> I can see this like: "cdrom": "/rhev/data-center/mnt/vserver-production.qa.lab.tlv.redhat.com:_iso__domain/d31037e5-0d45-4306-975b-73eea845ae86/images/11111111-1111-1111-1111-111111111111/CentOS-7-x86_64-NetInstall-1611.iso" 2. Then on hosts run the iptables command to block ISO Domain: iptables -I INPUT -s vserver-production.qa.lab.tlv.redhat.com -j DROP Actual Results: 1. engine reports that these VMs are not responding (and they appear with "?" mark in UI) 2018-01-16 23:02:38,135+02 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread- 15) [] EVENT_ID: VM_NOT_RESPONDING(126), VM golden_env_mixed_virtio_2_1 is not responding. 2. In <vdsm-client VM getStats> on host I see that "vmName": "golden_env_mixed_virtio_2_1", "status": "Up", is vdsmd log: INFO (libvirt/events) [virt.vm] (vmId='93d929d5-c60f-42ff-a248-e42206592d36') CPU stopped: onIOError (vm:5942) 2018-01-17 14:01:02,596+0200 WARN (check/loop) [storage.check] Checker u'/rhev/data-center/mnt/vserver-production.qa.lab.tlv.redhat.com:_iso__domain/d31037e5-0d45-4306-975b-73eea845ae86/dom_md/metadata' is blocked for 270.00 seconds (check:278) 3. the VM is in "Not Responding" state. In this state it is impossible neither to open console, nor ssh to the vm ip for investigate the logs and see I/O error. also in UI the IP disappears in this state. Engine&vdsm logs are attached as logs.tar.gz
(In reply to Polina from comment #42) > The bug was re-assigned because of the following tests results. > Test Steps: > 1. Run VM with CD attached from ISO domain. in <vdsm-client VM getStats> I > can see this like: > "cdrom": > "/rhev/data-center/mnt/vserver-production.qa.lab.tlv.redhat.com:_iso__domain/ > d31037e5-0d45-4306-975b-73eea845ae86/images/11111111-1111-1111-1111- > 111111111111/CentOS-7-x86_64-NetInstall-1611.iso" > > 2. Then on hosts run the iptables command to block ISO Domain: iptables -I > INPUT -s vserver-production.qa.lab.tlv.redhat.com -j DROP OK, but only access to ISO mount should be blocked. Where are the disk images served from? Are by any chance both data and iso domains residing on the same host, or does vserver-production.qa.lab.tlv.redhat.com serve only ISO domain? maybe try blocking only traffic from port 2049 (NFS)?
Created attachment 1383874 [details] engine, vdsm and rest files
Hi, I attache engine , vdsm logs and also rest responses illustrating the environment. the boot disk sits on iscsi - xtremio-iscsi1.scl.lab.tlv.redhat.com and the iso domain is on vserver-production.qa.lab.tlv.redhat.com. Please let me know if you need some additional info
please see comment #c48 and provide the info .
UpdateVolumes() periodic check seems to get stuck. So perhaps the comment there is relevant "# TODO: If this blocks (is it actually possible?)". Nir, thoughts? Polina, in the attached log I do not see any indication of VM() getting EIO (being paused) which may still have couple of reasons and the logs doesn't give enough information to conclude why. The log doesn't cover VM creation so I can't really verify the CDROM was started with error_policy=report. I see in the dumpxmls there's no error_policy set for CDROM and there's "stop" for the disk.
(In reply to Michal Skrivanek from comment #50) > UpdateVolumes() periodic check seems to get stuck. > So perhaps the comment there is relevant "# TODO: If this blocks (is it > actually possible?)". Nir, thoughts? Sure this can block, it calls self.cif.irs.getVolumeSize(), which can block up to 60 seconds on non-responsive NFS server. We do the blocking calls in ioprocess and the call will time out after 60 seconds, even if ioprocess is till blocked on storage. The current code may block the periodic thread pool because storage is not responsive. We should replace it with checks running in storage domain thread, submitting events about disks in this storage domain, so we never block virt threads - same way we do with storage domain monitoring. I think the consumer of this checks is storage code in engine updating disk actual size, using virt monitoring to get the value. In the virt layer we don't use or need the volume size.
(In reply to Polina from comment #48) > Hi, I attache engine , vdsm logs and also rest responses illustrating the > environment. > the boot disk sits on iscsi - xtremio-iscsi1.scl.lab.tlv.redhat.com > and the iso domain is on vserver-production.qa.lab.tlv.redhat.com. > Please let me know if you need some additional info problem is, Vdsm is still too old. The fix (I850d279f24c3f2ad401f7f93dd31bc29abdaa873) landed in Vdsm 4.20.14, but in the logs I see: 2018-01-21 09:23:55,499+0200 INFO (MainThread) [vds] (PID: 29848) I am the actual vdsm 4.20.13-1.el7ev cougar03.scl.lab.tlv.redhat.com (3.10.0-693.15.1.el7.x86_64) (vdsmd:148) So the error_policy values for both drives is incorrect (meaning, don't match the test expectations) <disk type=\'file\' device=\'cdrom\'> <driver name=\'qemu\' type=\'raw\'/> <source file=SNIP/CentOS-7-x86_64-NetInstall-1611.iso\'/> <backingStore/> <target dev=\'hdc\' bus=\'ide\'/> <readonly/> <alias name=\'ide0-1-0\'/> <address type=\'drive\' controller=\'0\' bus=\'1\' target=\'0\' unit=\'0\'/> </disk> <disk type=\'block\' device=\'disk\' snapshot=\'no\'> <driver name=\'qemu\' type=\'qcow2\' cache=\'none\' error_policy=\'stop\' io=\'native\'/> <source dev=\'SNIP\'/> <backingStore type=\'block\' index=\'1\'> <format type=\'qcow2\'/> <source dev=\'SNIP'/> <backingStore/> </backingStore> <target dev=\'vda\' bus=\'virtio\'/> <serial>c801010c-9530-4368-9860-42e440789978</serial> <boot order=\'1\'/> <alias name=\'virtio-disk0\'/> <address type=\'pci\' domain=\'0x0000\' bus=\'0x00\' slot=\'0x07\' function=\'0x0\'/> </disk> to make sure the test conditions are met: 1. make sure the XML Engine sent contains "error_policy=report" for the cdrom drive (this was the initial Engine fix) 2. make sure the XML Vdsm sends to Engine keeps the same value (here it was the Vdsm bug I fixed) Once both conditions above are met our job is basically done, but we can go ahead testing by disconnecting the storage just like you did.
Created attachment 1387181 [details] engine &vdsm 4.2.1-6 build. vdsm-4.20.17
Hi, I've tested on the last build http://bob.eng.lab.tlv.redhat.com/builds/4.2/rhv-4.2.1-6/rhv-release-4.2.1-6-001.noarch.rpm. in host : vdsm-common-4.20.17-1.el7ev.noarch vdsm-hook-openstacknet-4.20.17-1.el7ev.noarch vdsm-http-4.20.17-1.el7ev.noarch vdsm-hook-fcoe-4.20.17-1.el7ev.noarch vdsm-python-4.20.17-1.el7ev.noarch vdsm-hook-vmfex-dev-4.20.17-1.el7ev.noarch vdsm-client-4.20.17-1.el7ev.noarch vdsm-jsonrpc-4.20.17-1.el7ev.noarch vdsm-hook-ethtool-options-4.20.17-1.el7ev.noarch vdsm-yajsonrpc-4.20.17-1.el7ev.noarch vdsm-hook-vhostmd-4.20.17-1.el7ev.noarch vdsm-api-4.20.17-1.el7ev.noarch vdsm-4.20.17-1.el7ev.x86_64 vdsm-hook-vfio-mdev-4.20.17-1.el7ev.noarch vdsm-network-4.20.17-1.el7ev.x86_64 engine sends : <disk type="file" device="cdrom" snapshot="no"> <driver name="qemu" type="raw" error_policy="report"/> <source file="" startupPolicy="optional"/> <target dev="hdc" bus="ide"/> <readonly/> </disk> <disk snapshot="no" type="block" device="disk"> <target dev="vda" bus="virtio"/> <source dev="/rhev/data-center/mnt/blockSD/585d4e5d-2c4d-4c3a-9683-50db5d87a4cd/images/c8bf90c9-4638-484e-9488-16e8cb666f4c/3d8ab1aa-8e47-4153-ac2a-0f17af05f2f7"/> <driver name="qemu" io="native" type="qcow2" error_policy="stop" cache="none"/> <boot order="1"/> <serial>c8bf90c9-4638-484e-9488-16e8cb666f4c</serial> </disk> the same I see in vdsm.log - line 2524 attached new logs. it looks like the fix is not included in the last build.
(In reply to Polina from comment #55) > engine sends : > > <disk type="file" device="cdrom" snapshot="no"> > <target dev="hdc" bus="ide"/> > <driver name="qemu" type="raw" error_policy="report"/> > </disk> This is a cdrom "hdc"... > <disk snapshot="no" type="block" device="disk"> > <target dev="vda" bus="virtio"/> > <driver name="qemu" io="native" type="qcow2" error_policy="stop" > cache="none"/> > </disk> This is a disk "vda"... Please check the actual xml of the cdrom "hdc".
> (In reply to Nir Soffer from comment #56) > Please check the actual xml of the cdrom "hdc". from engine.log: <disk type="file" device="cdrom" snapshot="no"> <driver name="qemu" type="raw" error_policy="report"/> <source file="" startupPolicy="optional"/> <target dev="hdc" bus="ide"/> <readonly/> <address bus="1" controller="0" unit="0" type="drive" target="0"/> </disk>
(In reply to Polina from comment #55) > Hi, I've tested on the last build > http://bob.eng.lab.tlv.redhat.com/builds/4.2/rhv-4.2.1-6/rhv-release-4.2.1-6- > 001.noarch.rpm. > > in host : > vdsm-common-4.20.17-1.el7ev.noarch > vdsm-hook-openstacknet-4.20.17-1.el7ev.noarch > vdsm-http-4.20.17-1.el7ev.noarch > vdsm-hook-fcoe-4.20.17-1.el7ev.noarch > vdsm-python-4.20.17-1.el7ev.noarch > vdsm-hook-vmfex-dev-4.20.17-1.el7ev.noarch > vdsm-client-4.20.17-1.el7ev.noarch > vdsm-jsonrpc-4.20.17-1.el7ev.noarch > vdsm-hook-ethtool-options-4.20.17-1.el7ev.noarch > vdsm-yajsonrpc-4.20.17-1.el7ev.noarch > vdsm-hook-vhostmd-4.20.17-1.el7ev.noarch > vdsm-api-4.20.17-1.el7ev.noarch > vdsm-4.20.17-1.el7ev.x86_64 > vdsm-hook-vfio-mdev-4.20.17-1.el7ev.noarch > vdsm-network-4.20.17-1.el7ev.x86_64 > > engine sends : > > <disk type="file" device="cdrom" snapshot="no"> > <driver name="qemu" type="raw" error_policy="report"/> > <source file="" startupPolicy="optional"/> > <target dev="hdc" bus="ide"/> > <readonly/> > </disk> > <disk snapshot="no" type="block" device="disk"> > <target dev="vda" bus="virtio"/> > <source > dev="/rhev/data-center/mnt/blockSD/585d4e5d-2c4d-4c3a-9683-50db5d87a4cd/ > images/c8bf90c9-4638-484e-9488-16e8cb666f4c/3d8ab1aa-8e47-4153-ac2a- > 0f17af05f2f7"/> > <driver name="qemu" io="native" type="qcow2" error_policy="stop" > cache="none"/> > <boot order="1"/> > <serial>c8bf90c9-4638-484e-9488-16e8cb666f4c</serial> > </disk> > > the same I see in vdsm.log - line 2524 > attached new logs. > it looks like the fix is not included in the last build. Wait, why? I'm looking at vdsm.log in logs_build_4.2.1 and looks good, meaning that the test conditions seems fine now: We create the VM with the correct error policy: 2018-01-28 14:32:24,159+0200 INFO (jsonrpc/7) [api.virt] START create(vmParams={u'xml': u'<?xml version="1.0" encoding="UTF-8"?><domain type="kvm" xmlns:ovirt-tune="http://ovirt.org/vm/tune/1.0" xmlns:ovirt-vm="http://ovirt.org/vm/1.0"><name>my_test</name><uuid>81a9b3c1-a038-4b9c-96f2-00a5226b0b18 [...] Engine sends: <disk type="file" device="cdrom" snapshot="no"><driver name="qemu" type="raw" error_policy="report"></driver><source file="" startupPolicy="optional"></source><target dev="hdc" bus="ide"></target><readonly></readonly></disk> Vdsm correctly relays the configurations to libvirt: 2018-01-28 14:32:27,526+0200 INFO (vm/81a9b3c1) [virt.vm] (vmId='81a9b3c1-a038-4b9c-96f2-00a5226b0b18') <?xml version='1.0' encoding='utf-8'?> [...] <disk device="cdrom" snapshot="no" type="file"> <source file="" startupPolicy="optional" /> <target bus="ide" dev="hdc" /> <readonly /> <driver error_policy="report" io="threads" name="qemu" type="raw" /> </disk> And the configuration is further confirmed correct by the output of the 'dumpxmls' calls: 2018-01-28 14:32:32,029+0200 INFO (jsonrpc/3) [api.host] FINISH dumpxmls return={'status': {'message': 'Done', 'code': 0}, 'domxmls': {u'81a9b3c1-a038-4b9c-96f2-00a5226b0b18': '<domain type=\'kvm\' [...] <disk type=\'file\' device=\'cdrom\'>\n <driver name=\'qemu\' type=\'raw\' error_policy=\'report\' io=\'threads\'/>\n <source startupPolicy=\'optional\'/>\n <backingStore/>\n <target dev=\'hdc\' bus=\'ide\'/>\n <readonly/>\n <alias name=\'ide0-1-0\'/>\n <address type=\'drive\' controller=\'0\' bus=\'1\' target=\'0\' unit=\'0\'/>\n </disk> Please note that "error_policy=report" must be applied *only* to cdroms, not to disk devices. So now the test conditions are met. Again, it is good to test this scenario end to end, but now we are sending the correct configuration to libvirt, so if we see any bugs from now on is either another incorrect test scenario or libvirt/qemu issue. If you go ahead and disconnect storage to verify the behaviour, make sure that *only* the connection to cdrom storage (iso domain) is lost, disks should still be accessible. This is to minimize the noise in the test environment.
(In reply to Francesco Romani from comment #58) > If you go ahead and disconnect storage to verify the behaviour, make sure > that *only* the connection to cdrom storage (iso domain) is lost, disks > should still be accessible. I disconnect the only iso_domain connection to break cdrom. Please see below xml responce to https://{{host}}/ovirt-engine/api/storageconnections request. A minute after this block the VM is not responding (VM '81a9b3c1-a038-4b9c-96f2-00a5226b0b18'(my_test) moved from 'Up' --> 'NotResponding') If the right settings for error_policy mean the bug is fixed, and we still see the buggy behavior, maybe the bug must be moved to another team. But it could not be verified. <?xml version="1.0" encoding="UTF-8" standalone="yes"?> <storage_connections> <storage_connection href="/ovirt-engine/api/storageconnections/426678fa-5bed-414c-a691-c11f0a37d566" id="426678fa-5bed-414c-a691-c11f0a37d566"> <address>compute-ge-3.scl.lab.tlv.redhat.com</address> <path>/var/lib/exports/iso</path> <type>nfs</type> </storage_connection> <storage_connection href="/ovirt-engine/api/storageconnections/8587d106-5c0c-413c-aa3d-2f00d1bbeba7" id="8587d106-5c0c-413c-aa3d-2f00d1bbeba7"> <address>xtremio-iscsi1.scl.lab.tlv.redhat.com</address> <port>3260</port> <target>iqn.2008-05.com.xtremio:xio00153500071-514f0c50023f6c00</target> <type>iscsi</type> </storage_connection> <storage_connection href="/ovirt-engine/api/storageconnections/967f1fd1-3488-4724-8ad7-ee6941d873f2" id="967f1fd1-3488-4724-8ad7-ee6941d873f2"> <address>yellow-vdsb.qa.lab.tlv.redhat.com</address> <path>/Compute_NFS/GE/compute-ge-3/nfs_2</path> <type>nfs</type> </storage_connection> <storage_connection href="/ovirt-engine/api/storageconnections/2d8e1646-3d5e-44c8-9935-e43bc26a6a17" id="2d8e1646-3d5e-44c8-9935-e43bc26a6a17"> <address>yellow-vdsb.qa.lab.tlv.redhat.com</address> <path>/Compute_NFS/GE/compute-ge-3/nfs_1</path> <type>nfs</type> </storage_connection> <storage_connection href="/ovirt-engine/api/storageconnections/46407c1a-505f-4eab-9c83-caf6ce947b3a" id="46407c1a-505f-4eab-9c83-caf6ce947b3a"> <address>yellow-vdsb.qa.lab.tlv.redhat.com</address> <path>/Compute_NFS/GE/compute-ge-3/nfs_0</path> <type>nfs</type> </storage_connection> <storage_connection href="/ovirt-engine/api/storageconnections/d3034bc0-4327-4307-8756-deca33a2e9ae" id="d3034bc0-4327-4307-8756-deca33a2e9ae"> <address>yellow-vdsb.qa.lab.tlv.redhat.com</address> <path>/Compute_NFS/GE/compute-ge-3/export_domain</path> <type>nfs</type> </storage_connection> <storage_connection href="/ovirt-engine/api/storageconnections/1fc2b2ce-58b9-4f26-ac2a-1da8f2c712d5" id="1fc2b2ce-58b9-4f26-ac2a-1da8f2c712d5"> <address>vserver-production.qa.lab.tlv.redhat.com</address> <path>/iso_domain</path> <type>nfs</type> </storage_connection> <storage_connection href="/ovirt-engine/api/storageconnections/6a4b56b2-519c-46b2-aab4-2e0dd31a756a" id="6a4b56b2-519c-46b2-aab4-2e0dd31a756a"> <address>gluster01.scl.lab.tlv.redhat.com</address> <path>/compute-ge-3-volume3</path> <type>glusterfs</type> <vfs_type>glusterfs</vfs_type> </storage_connection> <storage_connection href="/ovirt-engine/api/storageconnections/e4231df6-7626-44f7-9b85-a032d30d3ece" id="e4231df6-7626-44f7-9b85-a032d30d3ece"> <address>gluster01.scl.lab.tlv.redhat.com</address> <path>/compute-ge-3-volume1</path> <type>glusterfs</type> <vfs_type>glusterfs</vfs_type> </storage_connection> <storage_connection href="/ovirt-engine/api/storageconnections/f0dda39f-0456-40e0-ade4-e8bf5b4616a6" id="f0dda39f-0456-40e0-ade4-e8bf5b4616a6"> <address>gluster01.scl.lab.tlv.redhat.com</address> <path>/compute-ge-3-volume2</path> <type>glusterfs</type> <vfs_type>glusterfs</vfs_type> </storage_connection> </storage_connections>
OK, we need a minimal reproducer running a VM with cdrom on NFS and another disk somewhere else. Let's run this reproducer using libvirt (without oVirt or even just vdsm) and move this down to libvirt developers.
I think what happens is this: 1. Blocking access to iso domain 2. Vdsm periodic check try to get all stats, including block stats 3. Libvirt try to get the size of all drives, including the cdrom 4. Qemu try to call stat() on the cdrom file and get stuck 5. Libvirt stuck waiting for qemu, VM becomes non-responsive Having "report" error policy will make qemu report an error to the guest when guest io fail trying to read from the cdrom. Since the storage is non responsive this will be reported after many minutes.
(In reply to Nir Soffer from comment #61) > I think what happens is this: > > 1. Blocking access to iso domain > 2. Vdsm periodic check try to get all stats, including block stats > 3. Libvirt try to get the size of all drives, including the cdrom > 4. Qemu try to call stat() on the cdrom file and get stuck > 5. Libvirt stuck waiting for qemu, VM becomes non-responsive > > Having "report" error policy will make qemu report an error to the guest when > guest io fail trying to read from the cdrom. Since the storage is non > responsive > this will be reported after many minutes. This could indeed explain the behaviour - let's hold on a bit more before to engage the libvirt developers. But could actually be worse than that, need to check if in the bulk stats flow libvirt does additional logic to guard against rogue storage - we had few bugs related to that for NFS storage, and few fixes back in time. Any way, I still think that despite this followup investigation is worth be done, once we send the correct error_policy to cdroms, this one BZ is fixed, and we need to continue on a new one.
(In reply to Francesco Romani from comment #62) I agree, handling issues with cdrom on NFS storage is not in the scope of this bug. We probably see the effect of "report" by testing cdrom on block storage (by uploading iso to disk on iSCSI or FC storage domain). With block storage our multipath configuration ensures that we fail fast (in 20-30 seconds) after access to storage is blocked.
Up until bug 1530730 all CDROMs were on NFS, so without fixing comment #52 we can't really say reporting works. I do not expect people moving away from the iso domain that fast Do we need UpdateVolumes() for CDROMs?
(In reply to Michal Skrivanek from comment #64) > Up until bug 1530730 all CDROMs were on NFS, so without fixing comment #52 > we can't really say reporting works. reporting works, but it cannot solve the issue of guest hang on non-responsive iso storage domain. > I do not expect people moving away from > the iso domain that fast I don't think it is related, when the storage becomes responsive, the guest should not stop but report the error back to the guest. > Do we need UpdateVolumes() for CDROMs? I see no reason to query cdroms size, they are readonly. Currently we are updating all drives: 379 def _execute(self): 380 for drive in self._vm.getDiskDevices(): 381 # TODO: If this blocks (is it actually possible?) 382 # we must make sure we don't overwrite good data 383 # with stale old data. 384 self._vm.updateDriveVolume(drive) We can skip readonly drives - hopefully engine will be ok with this. Tal, is engine ok with not having size info for readonly disks like cdroms?
(In reply to Nir Soffer from comment #65) > > I do not expect people moving away from > > the iso domain that fast > > I don't think it is related, when the storage becomes responsive, the guest > should > not stop but report the error back to the guest. to clarify my previous point - based on the logs it seems the monitoring _is_ the reason for hang. Policy seems to be est correctly to "report" and the only trouble seems to be with vdsm periodic polling
Tal? btw to change that for readonly drives would be enough for this BZ, but in general error_policy=report can be set on any drive, so eventually we need to handle a stuck updateDriveVolume()
error_policy should stay on 'guest' for CDROM drives, I don't mind particularly if the updateDriveVolume will not run on readonly disks as their size as static anyway
Verified in this test run: https://polarion.engineering.redhat.com/polarion/#/project/RHEVM3/testrun?id=4_2_IOErrorsISODom&tab=records&result=passed
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2018:1488
BZ<2>Jira Resync