Bug 886416
| Summary: | VMs are not migrated properly when NFS storage is blocked on host | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Virtualization Manager | Reporter: | Pavel Stehlik <pstehlik> | ||||
| Component: | ovirt-engine | Assignee: | Roy Golan <rgolan> | ||||
| Status: | CLOSED CURRENTRELEASE | QA Contact: | Ilanit Stein <istein> | ||||
| Severity: | high | Docs Contact: | |||||
| Priority: | unspecified | ||||||
| Version: | 3.1.0 | CC: | acathrow, dallan, dron, eedri, iheim, istein, jkt, juzhang, lpeer, mavital, michal.skrivanek, mkletzan, ofrenkel, rgolan, Rhev-m-bugs, vfeenstr, yeylon | ||||
| Target Milestone: | --- | ||||||
| Target Release: | 3.3.0 | ||||||
| Hardware: | All | ||||||
| OS: | Linux | ||||||
| Whiteboard: | virt | ||||||
| Fixed In Version: | is25 | Doc Type: | Bug Fix | ||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | Type: | Bug | |||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Bug Depends On: | 972675, 1045833 | ||||||
| Bug Blocks: | 1038284 | ||||||
| Attachments: |
|
||||||
|
Description
Pavel Stehlik
2012-12-12 09:10:39 UTC
*** Bug 851837 has been marked as a duplicate of this bug. *** (In reply to comment #0) > Expected results: > All VMs should migrate. Per KVM team in other BZ, VM should not be migrated if already paused on EIO. So the limit of concurrent migrations to 5 + the logic we added to block migration of paused VMs may lead to the scenario abov (In reply to comment #2) > (In reply to comment #0) > > > > Expected results: > > All VMs should migrate. > > Per KVM team in other BZ, VM should not be migrated if already paused on EIO. > So the limit of concurrent migrations to 5 + the logic we added to block > migration of paused VMs may lead to the scenario abov PS, my problem is more with the VMs that migrated as PAUSED - continuing these may lead to image corruption according to Dor. (In reply to comment #3) > (In reply to comment #2) > > (In reply to comment #0) > > > > > > > Expected results: > > > All VMs should migrate. > > > > Per KVM team in other BZ, VM should not be migrated if already paused on EIO. > > So the limit of concurrent migrations to 5 + the logic we added to block > > migration of paused VMs may lead to the scenario abov > > PS, my problem is more with the VMs that migrated as PAUSED - continuing > these may lead to image corruption according to Dor. Engine doesn't migrate PAUSED VMs and Pavel stated the he saw the VMs in MigratingFrom. could it be that VMs that didn't write to the disk started migrating and after the migration they tried and then stopped. (In reply to comment #4) > (In reply to comment #3) > > (In reply to comment #2) > > > (In reply to comment #0) 0> > PS, my problem is more with the VMs that migrated as PAUSED - continuing > > these may lead to image corruption according to Dor. > > Engine doesn't migrate PAUSED VMs and Pavel stated the he saw the VMs in > MigratingFrom. > could it be that VMs that didn't write to the disk started migrating and > after the migration they tried and then stopped. I suspect it could, and we may want to fail the migration in this case and leave the source VM in a pause state. If this is the case then this can only be done at the VDSM level. Michal, Andy your take on this? Moving to vdsm would probably help a bit, but doesn't solve the problem completely. we would need libvirt to fail this. Because even when at vdsm level the src VM can be healty, we issue migrate, and during the migration the src VM gets to EIO. Right now libvirt will complete the migration nevertheless Dave? Simon, is it even worth it to try to migrate away? It would always be a question of luck if the VM did access the storage or not before the migration finishes. IMHO not very predictable behavior. (In reply to comment #6) > Moving to vdsm would probably help a bit, but doesn't solve the problem > completely. > > > Simon, > is it even worth it to try to migrate away? It would always be a question of > luck if the VM did access the storage or not before the migration finishes. > IMHO not very predictable behavior. If the VM is already in EIO Pause - no Now, you probably mean is it worth it to try to migrate away on non-operational host that was caused by storage connectivity. The answer to the above is not strait forward there will be votes for any decisions we may take (This is why we added the cluster policy of 'do not auto migrate'. However since we may encounter storage disconnect even for VMs that have started migration from any other reason, then we need to handle the general case anyhow. related to qemu issue discussed in BZ#665820 no easy way how to avoid it as libvirt suffers from the same as rhev-m engine one idea is to check the status of the src vm once the last RAM page is transferred and src CPU is paused - but before the it is destroyed. If we get this far without EIO/ENOSPACE it's safe to destroy the src and complete migration, otherwise the dst needs to be abandoned Hi Michal,
I'm suggesting immediately pausing all domains affected by the loss of connectivity to the storage. That way you will keep the number of non-migrated domains at minimum.
In that case, the only domains ending in error would be those that started I/O after the storage was disconnected, but before oVirt noticed that (the pause was issued). I'm looking into modifying libvirt in a way that would help oVirt decide whether the migration is safe or not (without flooding libvirt with unnecessary API calls).
The time we get EIO from QEMU is a little non-deterministic from my POV, but I'm pretty sure that when the above problem appears (guest performs I/O on detached storage before the pause is issued), it is easy to know that it was paused due to that fact and not just because of the request. This information is saved along with the domain status ('virsh domstate <domain> --reason' reports that, for example).
Is that understandable?
(In reply to comment #10) > Hi Michal, > > I'm suggesting immediately pausing all domains affected by the loss of > connectivity to the storage. That way you will keep the number of > non-migrated domains at minimum. sound aggressive because your pausing also VMs that doesn't do I/O on that particular domain. so every storage disruption will pause the VMs unnecessarily. > > In that case, the only domains ending in error would be those that started > I/O after the storage was disconnected, but before oVirt noticed that (the > pause was issued). I'm looking into modifying libvirt in a way that would > help oVirt decide whether the migration is safe or not (without flooding > libvirt with unnecessary API calls). > > The time we get EIO from QEMU is a little non-deterministic from my POV, but > I'm pretty sure that when the above problem appears (guest performs I/O on > detached storage before the pause is issued), it is easy to know that it was > paused due to that fact and not just because of the request. This > information is saved along with the domain status ('virsh domstate <domain> > --reason' reports that, for example). > > Is that understandable? (In reply to comment #11) I don't feel it's aggressive at all. Of course you are pausing the machines that haven't touched NFS at all, but that's what you want. Let me explain in a different way. When host A is disconnected from the NFS share and starts migrating to host B, all the machines that were performing I/O on the NFS, while oVirt realized it's disconnected, are lost already (will be paused due to EIO). This we can't fix. Then there are the other machines that are OK and you start migrating them immediately, but not pause them. Since they are not paused, nobody can assure you they won't start any communication with storage before they are completely migrated away (unless QEMU adds something like "pause_only_storage" command), so these look OK, but we can lose them anyway. However, if you pause even those machines that don't perform any I/O on the disconnected storage, you can be sure there will be no other machine lost due the storage disconnection. (In reply to comment #12) > (In reply to comment #11) > I don't feel it's aggressive at all. Of course you are pausing the machines > that haven't touched NFS at all, but that's what you want. Let me explain > in a different way. > > When host A is disconnected from the NFS share and starts migrating to host > B, all the machines that were performing I/O on the NFS, while oVirt > realized it's disconnected, are lost already (will be paused due to EIO). > This we can't fix. Then there are the other machines that are OK and you > start migrating them immediately, but not pause them. Since they are not > paused, nobody can assure you they won't start any communication with > storage before they are completely migrated away (unless QEMU adds something > like "pause_only_storage" command), so these look OK, but we can lose them > anyway. However, if you pause even those machines that don't perform any > I/O on the disconnected storage, you can be sure there will be no other > machine lost due the storage disconnection. Your point is clear, thanks. worth a policy based decision in that case because what if the I/O problem is local to the host itself? by migrating the VMs you save them from pausing. Ovirt can set a host to be non-operational if it detects the host has latency in I/O on its storage domains so based on that we would like to complete the migration to the dest. I maybe missing something though - Simon? was referring to Resilience Policy at Cluster level (In reply to comment #13) > I maybe missing something though - Simon? We need to think here on all possible scenarios. I tend to agree that if a storage domain is detected as unresponsive we should immediately PAUSE all the VMs that are using it - this probably should be done even on the VDSM level. Those machine, if not PAUSED on EIO, should be safe to migrate. This way you may end up with some VM that are paused which may have otherwise continued, but with many others VMs that where practically saved since they where properly paused before getting hit, and now may be safely migrated and continue to run on a different host - that is if the failure is local. this needs minor code change and proper testing once bug 972675 gets in, hence proposing exception see bug 961154 for the corresponding vdsm change. Here we need on engine side at least a VdcOptions config for 3.3 clusters to send a new flag Michal, Can you please advise how to verify this bug? I tried to block storage (iptables -A OUTPUT -d <storage server> -j REJECT) on source host, VM event "VM not responding" appeared, then VM changed from up to "?", then it was migrated (automatically), and on destination it was in status pause. That's wrong...can you please look at that with Vinzenz. It should not appear as Paused on destination. Pls check the abort on eio option (3.3 cluster level engine-config option) abort on eio option: # engine-config -g AbortMigrationOnError AbortMigrationOnError: true version: 3.3 AbortMigrationOnError: false version: 3.2 AbortMigrationOnError: false version: 3.1 AbortMigrationOnError: false version: 3.0 Also on source host vdsm.log this flag is seen: 'abortOnError': 'true' Pete mentioned in his patch that libvirt supports this flag since version 1.0.1 upstream I checked cyan-vdse.qa.lab and its libvirt version is 0.10.2 (In reply to Roy Golan from comment #22) > Pete mentioned in his patch that libvirt supports this flag since version > 1.0.1 upstream > > I checked cyan-vdse.qa.lab and its libvirt version is 0.10.2 The question is if this is RHEL, which by the version it looks like it is, contains this feature as a backport. Unfortunately you can't go by upstream versions with features when it comes to RHEL thanks Eyal. In order to verify this 3.3 bug, which deals with backend add abortOnError flag, I first need to know that libvirt (libvirt-0.10.2-29.el6.1.x86_64) supports this flag - where can I find this info on downstream libvirt please? I see that this seems not to be working. When I am dropping all packets from the storage server on the source host side then the migration still continues, however the VM gets paused on the source. However the VM migration does not get finalized and stays stuck at about 99% and eventually should get aborted due to our no-progress timeout, however even that does not happen. If I allow then the connection to the storage server again suddenly we get a bunch of events and surprisingly even though we called abortJob the migration suceeds and the VM is destination on the target, and eventually happened to run again on the destination. To me it looks like that libvirt got stuck because of the EIO, however the abortonerror flag definitely does NOT work. These are the events I was referring to: Thread-197::DEBUG::2013-12-17 16:24:10,112::fileSD::154::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.64.102:_volumes_wolf_data__0__nfs__2013082717821706028/7ac99a5a-1542-4757-b7fc-c43ac2caf8f4 libvirtEventLoop::INFO::2013-12-17 16:24:10,112::vm::4366::vm.Vm::(_onAbnormalStop) vmId=`f7afb5e9-9f6a-4cca-8ef5-7d02e9a76969`::abnormal vm stop device virtio-disk0 error eio libvirtEventLoop::INFO::2013-12-17 16:24:10,120::vm::4366::vm.Vm::(_onAbnormalStop) vmId=`f7afb5e9-9f6a-4cca-8ef5-7d02e9a76969`::abnormal vm stop device virtio-disk0 error eio Thread-122::DEBUG::2013-12-17 16:24:10,122::fileSD::154::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.64.102:_volumes_wolf_export__istein__0__nfs__2013110512489493611/82d5b7d4-100b-4073-9496-cc3810c24168 libvirtEventLoop::INFO::2013-12-17 16:24:10,123::vm::4366::vm.Vm::(_onAbnormalStop) vmId=`f7afb5e9-9f6a-4cca-8ef5-7d02e9a76969`::abnormal vm stop device virtio-disk0 error eio libvirtEventLoop::INFO::2013-12-17 16:24:10,129::vm::4366::vm.Vm::(_onAbnormalStop) vmId=`f7afb5e9-9f6a-4cca-8ef5-7d02e9a76969`::abnormal vm stop device virtio-disk0 error eio libvirtEventLoop::INFO::2013-12-17 16:24:10,130::vm::4366::vm.Vm::(_onAbnormalStop) vmId=`f7afb5e9-9f6a-4cca-8ef5-7d02e9a76969`::abnormal vm stop device virtio-disk0 error eio libvirtEventLoop::INFO::2013-12-17 16:24:10,130::vm::4366::vm.Vm::(_onAbnormalStop) vmId=`f7afb5e9-9f6a-4cca-8ef5-7d02e9a76969`::abnormal vm stop device virtio-disk0 error eio libvirtEventLoop::INFO::2013-12-17 16:24:10,130::vm::4366::vm.Vm::(_onAbnormalStop) vmId=`f7afb5e9-9f6a-4cca-8ef5-7d02e9a76969`::abnormal vm stop device virtio-disk0 error eio libvirtEventLoop::INFO::2013-12-17 16:24:10,130::vm::4366::vm.Vm::(_onAbnormalStop) vmId=`f7afb5e9-9f6a-4cca-8ef5-7d02e9a76969`::abnormal vm stop device virtio-disk0 error eio libvirtEventLoop::INFO::2013-12-17 16:24:10,130::vm::4366::vm.Vm::(_onAbnormalStop) vmId=`f7afb5e9-9f6a-4cca-8ef5-7d02e9a76969`::abnormal vm stop device virtio-disk0 error eio libvirtEventLoop::INFO::2013-12-17 16:24:10,131::vm::4366::vm.Vm::(_onAbnormalStop) vmId=`f7afb5e9-9f6a-4cca-8ef5-7d02e9a76969`::abnormal vm stop device virtio-disk0 error eio libvirtEventLoop::INFO::2013-12-17 16:24:10,131::vm::4366::vm.Vm::(_onAbnormalStop) vmId=`f7afb5e9-9f6a-4cca-8ef5-7d02e9a76969`::abnormal vm stop device virtio-disk0 error eio libvirtEventLoop::INFO::2013-12-17 16:24:10,131::vm::4366::vm.Vm::(_onAbnormalStop) vmId=`f7afb5e9-9f6a-4cca-8ef5-7d02e9a76969`::abnormal vm stop device virtio-disk0 error eio libvirtEventLoop::INFO::2013-12-17 16:24:10,131::vm::4366::vm.Vm::(_onAbnormalStop) vmId=`f7afb5e9-9f6a-4cca-8ef5-7d02e9a76969`::abnormal vm stop device virtio-disk0 error eio libvirtEventLoop::INFO::2013-12-17 16:24:10,132::vm::4366::vm.Vm::(_onAbnormalStop) vmId=`f7afb5e9-9f6a-4cca-8ef5-7d02e9a76969`::abnormal vm stop device virtio-disk0 error eio libvirtEventLoop::INFO::2013-12-17 16:24:10,132::vm::4366::vm.Vm::(_onAbnormalStop) vmId=`f7afb5e9-9f6a-4cca-8ef5-7d02e9a76969`::abnormal vm stop device virtio-disk0 error eio libvirtEventLoop::INFO::2013-12-17 16:24:10,132::vm::4366::vm.Vm::(_onAbnormalStop) vmId=`f7afb5e9-9f6a-4cca-8ef5-7d02e9a76969`::abnormal vm stop device virtio-disk0 error eio libvirtEventLoop::INFO::2013-12-17 16:24:10,132::vm::4366::vm.Vm::(_onAbnormalStop) vmId=`f7afb5e9-9f6a-4cca-8ef5-7d02e9a76969`::abnormal vm stop device virtio-disk0 error eio libvirtEventLoop::INFO::2013-12-17 16:24:10,133::vm::4366::vm.Vm::(_onAbnormalStop) vmId=`f7afb5e9-9f6a-4cca-8ef5-7d02e9a76969`::abnormal vm stop device virtio-disk0 error eio libvirtEventLoop::INFO::2013-12-17 16:24:10,133::vm::4366::vm.Vm::(_onAbnormalStop) vmId=`f7afb5e9-9f6a-4cca-8ef5-7d02e9a76969`::abnormal vm stop device virtio-disk0 error eio libvirtEventLoop::INFO::2013-12-17 16:24:10,133::vm::4366::vm.Vm::(_onAbnormalStop) vmId=`f7afb5e9-9f6a-4cca-8ef5-7d02e9a76969`::abnormal vm stop device virtio-disk0 error eio libvirtEventLoop::INFO::2013-12-17 16:24:10,133::vm::4366::vm.Vm::(_onAbnormalStop) vmId=`f7afb5e9-9f6a-4cca-8ef5-7d02e9a76969`::abnormal vm stop device virtio-disk0 error eio libvirtEventLoop::INFO::2013-12-17 16:24:10,133::vm::4366::vm.Vm::(_onAbnormalStop) vmId=`f7afb5e9-9f6a-4cca-8ef5-7d02e9a76969`::abnormal vm stop device virtio-disk0 error eio libvirtEventLoop::INFO::2013-12-17 16:24:10,134::vm::4366::vm.Vm::(_onAbnormalStop) vmId=`f7afb5e9-9f6a-4cca-8ef5-7d02e9a76969`::abnormal vm stop device virtio-disk0 error eio libvirtEventLoop::INFO::2013-12-17 16:24:10,134::vm::4366::vm.Vm::(_onAbnormalStop) vmId=`f7afb5e9-9f6a-4cca-8ef5-7d02e9a76969`::abnormal vm stop device virtio-disk0 error eio libvirtEventLoop::INFO::2013-12-17 16:24:10,134::vm::4366::vm.Vm::(_onAbnormalStop) vmId=`f7afb5e9-9f6a-4cca-8ef5-7d02e9a76969`::abnormal vm stop device virtio-disk0 error eio libvirtEventLoop::INFO::2013-12-17 16:24:10,134::vm::4366::vm.Vm::(_onAbnormalStop) vmId=`f7afb5e9-9f6a-4cca-8ef5-7d02e9a76969`::abnormal vm stop device virtio-disk0 error eio libvirtEventLoop::INFO::2013-12-17 16:24:10,135::vm::4366::vm.Vm::(_onAbnormalStop) vmId=`f7afb5e9-9f6a-4cca-8ef5-7d02e9a76969`::abnormal vm stop device virtio-disk0 error eio libvirtEventLoop::INFO::2013-12-17 16:24:10,135::vm::4366::vm.Vm::(_onAbnormalStop) vmId=`f7afb5e9-9f6a-4cca-8ef5-7d02e9a76969`::abnormal vm stop device virtio-disk0 error eio libvirtEventLoop::INFO::2013-12-17 16:24:10,135::vm::4366::vm.Vm::(_onAbnormalStop) vmId=`f7afb5e9-9f6a-4cca-8ef5-7d02e9a76969`::abnormal vm stop device virtio-disk0 error eio libvirtEventLoop::INFO::2013-12-17 16:24:10,135::vm::4366::vm.Vm::(_onAbnormalStop) vmId=`f7afb5e9-9f6a-4cca-8ef5-7d02e9a76969`::abnormal vm stop device virtio-disk0 error eio libvirtEventLoop::INFO::2013-12-17 16:24:10,135::vm::4366::vm.Vm::(_onAbnormalStop) vmId=`f7afb5e9-9f6a-4cca-8ef5-7d02e9a76969`::abnormal vm stop device virtio-disk0 error eio libvirtEventLoop::INFO::2013-12-17 16:24:10,136::vm::4366::vm.Vm::(_onAbnormalStop) vmId=`f7afb5e9-9f6a-4cca-8ef5-7d02e9a76969`::abnormal vm stop device virtio-disk0 error eio Moving this bug to verified, since flag 'abortOnError': 'true' is seen in vdsm.log Also putting depends on Bug 972675 (The related libvirt bug, that suppose to respond to this flag). Closing - RHEV 3.3 Released Closing - RHEV 3.3 Released |