Bug 1793132
Summary: | hostpath-provisioner - PV doesn't get removed after deleting DV (when attempting to run out of space) | |||
---|---|---|---|---|
Product: | Container Native Virtualization (CNV) | Reporter: | Alex Kalenyuk <akalenyu> | |
Component: | Documentation | Assignee: | Andrew Burden <aburden> | |
Status: | CLOSED CURRENTRELEASE | QA Contact: | Irina Gulina <igulina> | |
Severity: | high | Docs Contact: | ||
Priority: | high | |||
Version: | 2.2.0 | CC: | aburden, alitke, cnv-qe-bugs, ncredi, ngavrilo, rgarcia, sgordon, ycui | |
Target Milestone: | --- | |||
Target Release: | 2.2.0 | |||
Hardware: | Unspecified | |||
OS: | Unspecified | |||
Whiteboard: | ||||
Fixed In Version: | 2.2.0 | Doc Type: | Enhancement | |
Doc Text: |
Feature: HPP
Reason: see comment 4
Result: see comment 5
|
Story Points: | --- | |
Clone Of: | ||||
: | 1794050 (view as bug list) | Environment: | ||
Last Closed: | 2020-03-04 10:53:59 UTC | Type: | Bug | |
Regression: | --- | Mount Type: | --- | |
Documentation: | --- | CRM: | ||
Verified Versions: | Category: | --- | ||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
Cloudforms Team: | --- | Target Upstream Version: | ||
Embargoed: | ||||
Bug Depends On: | ||||
Bug Blocks: | 1794050 |
Description
Alex Kalenyuk
2020-01-20 18:13:23 UTC
We should have a release note once we have a workaround for this. Adam, could you provide the proper workaround when this bug happens? Alexander, can you attempt to reproduce this and determine the least invasive workaround for when this issue occurs on a CNV 2.2 installation? So the problem is that when we use /var/hpvolumes as the path, we are using the same partition as where the OS is running. So basically we are leaving the OS no temporary space that it uses for various things, and thus the entire node goes down. Once that happens the kubelet is dead, and nothing you do from a cluster perspective will cure it. You will have manually go into the node and free up some space so the OS and thus the kubelet can recover, after which everything should start working again. Solution: Make a separate partition for your storage needs that does not share storage with the OS. Opened a PR against the hpp, to have a warning in the README about not sharing the storage with the OS. https://github.com/kubevirt/hostpath-provisioner/pull/39 Alexander confirmed the workaround is to access the node and free up storage. I believe the next step is to engage the Documentation team to update the product documentation with a recommendation similar to the one given in https://github.com/kubevirt/hostpath-provisioner/pull/39. according to comment #6, moving this bug to doc team. And the cloned bug 1794050 in Storage component to fix the real issue in future release. Warning admonition added to two modules, after the prerequisite steps for 'Create a backing partition...` PR: https://github.com/openshift/openshift-docs/pull/19846/files Sorry, there's no preview build, but all affected content is in the following (not-updated) section: https://docs.openshift.com/container-platform/4.3/cnv/cnv_virtual_machines/cnv_virtual_disks/cnv-configuring-local-storage-for-vms.html Note, this change is only targeted to CNV 2.2, as bz#1794050 is fixing this for 2.3 Andrew, docs PR looks good to me, merge please. Thanks. This update was included in the most recent 4.3 build. It can be viewed here: https://docs.openshift.com/container-platform/4.3/cnv/cnv_virtual_machines/cnv_virtual_disks/cnv-configuring-local-storage-for-vms.html |