Bug 1992753
Summary: | virtctl start vm does not check whether the pvc within the VM is mounted in another pod and the vm starts | ||
---|---|---|---|
Product: | Container Native Virtualization (CNV) | Reporter: | Kevin Alon Goldblatt <kgoldbla> |
Component: | Storage | Assignee: | Alice Frosi <afrosi> |
Status: | CLOSED WONTFIX | QA Contact: | Kevin Alon Goldblatt <kgoldbla> |
Severity: | high | Docs Contact: | |
Priority: | high | ||
Version: | 4.9.0 | CC: | afrosi, alitke, cnv-qe-bugs, kbidarka, sgott, yadu |
Target Milestone: | --- | ||
Target Release: | 4.12.0 | ||
Hardware: | Unspecified | ||
OS: | Unspecified | ||
Whiteboard: | Doc: Remains a known issue in the 4.11 GA release notes | ||
Fixed In Version: | Doc Type: | Known Issue | |
Doc Text: |
In some instances, multiple virtual machines can mount the same PVC in read-write mode, which might result in data corruption. (BZ#1992753)
As a workaround, avoid using a single PVC in read-write mode with multiple VMs.
|
Story Points: | --- |
Clone Of: | Environment: | ||
Last Closed: | 2022-11-02 12:44:46 UTC | Type: | Bug |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: |
Description
Kevin Alon Goldblatt
2021-08-11 17:29:38 UTC
This is a known issue because we don't have a locking or protection mechanism for PVCs in Kubernetes. The problem is not that 2 pods access the PVC at the same time, but that 2 instances of QEMU could write at the same time the disk. QEMU offers partial protection as in certain cases the second instance fails to run because it cannot acquire the write lock. Kevin, you should try in your example to run again libguestfs after the VM has started. It should fail in this case because QEMU cannot acquire the lock. However, I've already been able to reproduce the case where QEMU was not able to detect the lock and start 2 QEMU instances with the same disk. In my setup, it was an RWX with ceph, and the 2 pods were scheduled on 2 different nodes. At least in my experience, QEMU has always have been able to detect the lock in the case the QEMU instances were running on the same node. Kubernetes is introducing a new access mode (ReadWriteOncePod) that prevents 2 pods to use the same PVC. This could prevent multiple pods to use the same PVC however it is a very restrictive mode and it prevents the VM to be migratable. In order to solve this, we need a proper solution to protect PVC at least in KubeVirt also when the mode is ReadWriteMany. Thanks Alice, What do you envision that KubeVirt would be able to do about this outside of what Kubernetes is already planning? This PR should fix the issue once merged: - https://github.com/kubevirt/kubevirt/pull/6362 Alice, It looks like https://github.com/kubevirt/kubevirt/pull/6362 was closed. Is this something you're actively working on? Stu, unfortunately, this requires some locking mechanism in Kubernetes or coordination from KubeVirt. I tried to introduce a partial control in KubeVirt in that PR but it was rejected. So, I still need to figure out a proper way. Deferring to the next release due to the anticpated complexity of fixing this. Added Release note > known issue In some instances, multiple virtual machines can mount the same PVC in read-write mode, which might result in data corruption. As a workaround, avoid using a single PVC in read-write mode with multiple VMs. (BZ#1992753) https://github.com/openshift/openshift-docs/pull/42530 https://deploy-preview-42530--osdocs.netlify.app/openshift-enterprise/latest/virt/virt-4-10-release-notes#virt-4-10-known-issues Future link: After the OpenShift Virtualization 4.10 releases, you can find the release notes here: https://docs.openshift.com/container-platform/4.10/virt/virt-4-10-release-notes.html or on the portal, https://access.redhat.com/documentation/en-us/openshift_container_platform/4.10 Moving this to 4.12, due to complexity of fixing this. Adam, This bug looks more related to storage component, dealing with PVC and ReadWrite access. Do you want to take the ownership of this bug and be moved to the Storage component? Yes, taking this into the Storage component. Alice, Can we introduce a check before starting the VM that the PVCs are not in use? I understand that this is racy but it may be better than nothing. Thoughts? Adam, yes I tried to do it in the PR mentioned above: https://github.com/kubevirt/kubevirt/pull/6362 but the solution was rejected upstream Our proposed solution has been rejected by the kubevirt maintainers so at this time it will not be possible to provide a fix for this issue. Closed - Won't fix. Per @apinnick. Reviewed on Jan 12, 2023, Leave this known issue in the 4.12 release notes because it's still a known issue and won't be fixed. |