Bug 857456
| Summary: | Alert detected on RHEV-M - VM has been paused due to a storage error. | ||
|---|---|---|---|
| Product: | [Red Hat Storage] Red Hat Gluster Storage | Reporter: | Gowrishankar Rajaiyan <grajaiya> |
| Component: | fuse | Assignee: | Brian Foster <bfoster> |
| Status: | CLOSED INSUFFICIENT_DATA | QA Contact: | Gowrishankar Rajaiyan <grajaiya> |
| Severity: | unspecified | Docs Contact: | |
| Priority: | high | ||
| Version: | unspecified | CC: | aavati, bfoster, csaba, grajaiya, iheim, rhs-bugs, shaines, vbellur, vinaraya |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2012-12-12 21:59:59 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
Gowrishankar Rajaiyan
2012-09-14 13:32:57 UTC
Skimming through the libvirt logs, I see something similar to an issue I saw when setting up our rhevm instance: qemu-kvm: -drive file=/rhev/data-center/609c0436-4b5c-4639-89ed-1fc60ccaa9a1/9c17fd91-2e28-463d-b9b3-93fcd9a77679/images/8179fad6-6fe8-4f19-9fdc-76592eef53d8/86c8e2c0-3a32-4f00-ad21-1c02afc304db,if=none,id=drive-virtio-disk0,format=raw,serial=8179fad6-6fe8-4f19-9fdc-76592eef53d8,cache=none,werror=stop,rerror=stop,aio=threads: could not open disk image /rhev/data-center/609c0436-4b5c-4639-89ed-1fc60ccaa9a1/9c17fd91-2e28-463d-b9b3-93fcd9a77679/images/8179fad6-6fe8-4f19-9fdc-76592eef53d8/86c8e2c0-3a32-4f00-ad21-1c02afc304db: Permission denied 2012-09-13 09:41:42.838+0000: shutting down I disabled selinux on hypervisor hosts to bypass a similar error. If I re-enable and try a migration to that host, however, the migration fails (as opposed to migrating the vm and stopping due to error). It sounds like it could be something different, but out of curiosity, is this a persistent state of the hypervisor host? In other words, can you forcibly start the paused VMs, or any VM on that host, or is the error transient during the migration? If persistent, does an 'echo 0 > /selinux/enforce' change anything? Can you please respond to Brian's questions? Unfortunately the setup no longer exists. However, the selinux error in comment #6 are similar to the one at https://bugzilla.redhat.com/show_bug.cgi?id=835936 and addressed in selinux-policy-3.7.19-155.el6_3.6 Taking another look at the libvirtd.log, I see snippets similar to that below for several VMs. This suggests a VM was paused due to a failed write (EACCES, "Permission denied"), but I'm not seeing anything that correlates this to a problem in gluster.
FWIW, I've been doing significant rhevm testing on a dhtrep2 volume the past couple of weeks (including doing migrations/self-heals) and I've yet to see a VM paused error as described here. I think we'll need more information to make further progress with this bug:
- Some kind of insight into the steps to reproduce.
- Ideally, access to an effected setup in this state to determine reproducibility, persistence of the error and potentially to debug.
--- libvirtd.log ---
qemuMonitorEmitIOError:954 : mon=0x7f49d81261d0
qemuMonitorRef:201 : QEMU_MONITOR_REF: mon=0x7f49d81261d0 refs=4
qemuProcessHandleIOError:854 : Transitioned guest VM_5 to paused state due to IO error
qemuProcessHandleIOError:864 : Preserving lock state '(null)'
qemuMonitorUnref:210 : QEMU_MONITOR_UNREF: mon=0x7f49d81261d0 refs=3
qemuMonitorUnref:210 : QEMU_MONITOR_UNREF: mon=0x7f49d81261d0 refs=2
virDomainFree:2313 : dom=0x97a350, (VM: name=VM_5, uuid=0e8a2034-f4b1-4d34-8859-47e35793373e)
virDomainFree:2313 : dom=0x97a350, (VM: name=VM_5, uuid=0e8a2034-f4b1-4d34-8859-47e35793373e)
qemuMonitorRef:201 : QEMU_MONITOR_REF: mon=0x7f49d81261d0 refs=3
qemuMonitorIOProcess:327 : QEMU_MONITOR_IO_PROCESS: mon=0x7f49d81261d0 buf={"timestamp": {"seconds": 1347628099, "microseconds": 478179}, "event": "STOP"}^M
{"timestamp": {"seconds": 1347628099, "microseconds": 478478}, "event": "BLOCK_IO_ERROR", "data": {"device": "drive-virtio-disk0", "__com.redhat_debug_info": {"message": "Permission denied", "errno": 13}, "__com.redhat_reason": "eother", "operation": "write", "action": "stop"}}
Setting this closed due to insufficient data. Please re-open if we can reproduce with more data and potential access to the configuration in this state. |