Bug 1496399
| Summary: | [downstream clone - 4.1.7] Shutdown of a vm during snapshot deletion renders the disk invalid | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Virtualization Manager | Reporter: | rhev-integ |
| Component: | ovirt-engine | Assignee: | Ala Hino <ahino> |
| Status: | CLOSED ERRATA | QA Contact: | Kevin Alon Goldblatt <kgoldbla> |
| Severity: | medium | Docs Contact: | |
| Priority: | high | ||
| Version: | 4.1.2 | CC: | ableisch, ahino, amureini, ebenahar, kgoldbla, lsurette, lveyde, ratamir, rbalakri, Rhev-m-bugs, srevivo, tnisan, ykaul, ylavi |
| Target Milestone: | ovirt-4.1.7 | Keywords: | ZStream |
| Target Release: | --- | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | ovirt-engine-4.1.7.4 | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | 1467928 | Environment: | |
| Last Closed: | 2017-11-07 17:27:54 UTC | Type: | --- |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | Storage | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | 1467928 | ||
| Bug Blocks: | 1384321 | ||
|
Description
rhev-integ
2017-09-27 09:53:29 UTC
Created attachment 1294646 [details]
vdsm.log from SPM-host
(Originally by Andreas Bleischwitz)
Hi Andreas, I am trying to reproduce this issue. I did the following: 1. created a VM with 4 disks 2. created a snapshot 3. copied data to each disk 4. deleted the snapshot 5. powered-off the VM while deleting the snapshot. The delete operation failed, the snapshot marked as OK and the status of each disk was illegal. All as expected and I am able to start the VM again and to delete the snapshot. Can you please elaborate what do you mean by disks marked as invalid? Where/how do you see that? (Originally by Ala Hino) Hi Ala, after my deletion of the snapshot was marked as failed, the disk within the snapshot was also marked as failed. See Virtual Machines -> [vm] -> Snapshots -> [snapshot] -> disks. Currently I have not been able to remove *any* of my snapshots from that test-machine. I also have not been able to start the VM as one of the disk have been reported as invalid (Bad volume specification). (Originally by Andreas Bleischwitz) Thanks Andreas. It seems that one of the delete operations succeeded, hence 33b202bd-55e7-4a0f-b6a1-b9057aee8099 doesn't exist. The attached Vdsm log is partial. Can you please upload full Vdsm log? (Originally by Ala Hino) Created attachment 1297436 [details]
vdsm.log part 1
(Originally by Andreas Bleischwitz)
Created attachment 1297437 [details]
vdsm.log part 2
(Originally by Andreas Bleischwitz)
Created attachment 1297438 [details]
vdsm.log part 3
(Originally by Andreas Bleischwitz)
Andreas, Can you please upload the SPM log as well? It seems that after the VM was shutdown, there was an attempt to delete the snapshot while the VM is down (aka cold merge), is this correct? If so, I'd like to ask you to try the flow again but this time without doing cold merge, and see whether it is possible to start the VM after it was shutdown during live merge. (Originally by Ala Hino) The engine log seems partial, can you please send the full log? (Originally by Ala Hino) Created attachment 1303777 [details]
engine.log
(Originally by Andreas Bleischwitz)
If you have the SPM log of the last failure and can upload it, that could be helpful further analyzing. (Originally by Ala Hino) Hi Ala, those logs have been rotated into nirvana unfortunately. I will append a complete set of logs after I have had time to re-produce that issue. Are there some more than engine.log and vdsm.log you would need? (Originally by Andreas Bleischwitz) Please remember to upload the logs of the SPM and the host running the VM, in addition to the engine. It would be very helpful if you can document every step you perform - number of disks created, number of snapshot created, the time you perform the shutdown - is it specific time or some random time? Also, the chain info (vdsm and qemu) would be useful - before the merge and after the shutdown. After the live merge and the shutdown, do you perform a cold merge? If yes, please try to run the VM *before* and after the cold merge, and send the chain info after the cold merge. (Originally by Ala Hino) Hi Andreas, Any news on this? (Originally by Ala Hino) Pushing to 4.1.6 until we're able to reproduce. (Originally by Allon Mureinik) This patch caused OST failures and was reverted. Moving back to ASSIGNED. Based on comment #27 - moving back to assigned New patches were merged after the original ones were reverted (check out the block between comment 27 and comment 28). Moving back to ON_QA Thanks Allon Verified with the following code: ----------------------------------------- ovirt-engine-4.1.7.4-0.1.el7.noarch vdsm-4.19.35-1.el7ev.x86_64 Verified with the following scenario: ----------------------------------------- Steps to reproduce: Steps to reproduce: 1. On a vm with 2 disks create a snapshot. 2. Add some data 3. Live merge the leaf's parent snapshot and watch the engine logs 4. When merge steps completes, block access from the host to the engine 5. At this point, the engine will keep retrying checking the merge status, and fail because there is no access to the host 6. Shutdown the VM - snapshot deletion completes successfully 7. Start the VM, works fine - no disks are left invalid Moving to VERIFIED! Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2017:3138 |