Bug 2038985 - No feedback when HPP path is sharing host filesystem
Summary: No feedback when HPP path is sharing host filesystem
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Container Native Virtualization (CNV)
Classification: Red Hat
Component: Storage
Version: 4.10.0
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: ---
: 4.10.0
Assignee: Alex Kalenyuk
QA Contact: Jenia Peimer
URL:
Whiteboard: release note
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-01-10 17:04 UTC by Alex Kalenyuk
Modified: 2022-03-16 16:06 UTC (History)
3 users (show)

Fixed In Version: hostpath-csi-driver-rhel8 v4.10.0-31, CNV v4.10.0-651
Doc Type: Release Note
Doc Text:
When using the hostpath provisioner (HPP) to provide storage for virtual machine disks, configure it with dedicated storage that is separate from the node's root filesystem. Otherwise, the node could run out of storage and become unschedulable. A new HPPSharingPoolPathWithOS alert now warns administrators if this misconfiguration is detected.
Clone Of:
Environment:
Last Closed: 2022-03-16 16:06:09 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
HPP Alert screenshot (222.44 KB, image/png)
2022-02-09 13:39 UTC, Jenia Peimer
no flags Details
HPP Alert screenshot 2 (198.30 KB, image/png)
2022-02-09 13:43 UTC, Jenia Peimer
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Github kubevirt hostpath-provisioner-operator pull 207 0 None Merged Make DaemonSet metrics scrapable 2022-01-20 09:46:22 UTC
Github kubevirt hostpath-provisioner-operator pull 215 0 None Merged [release-v0.12] Make DaemonSet metrics scrapable 2022-02-01 10:22:08 UTC
Github kubevirt hostpath-provisioner pull 100 0 None Merged Check if pool paths are shared with OS 2022-02-03 14:20:27 UTC
Github kubevirt hostpath-provisioner pull 105 0 None Merged [release-v0.12] Manual backport of #95, #100 for CI to pass 2022-02-03 15:21:40 UTC
Red Hat Product Errata RHSA-2022:0947 0 None Closed RHEL EUS Errata Documentation 2022-06-16 06:26:55 UTC

Description Alex Kalenyuk 2022-01-10 17:04:59 UTC
Description of problem:
When installing HPP, there is no feedback if the specified path is sharing a filesystem with other critical components (kubelet/OS)

Version-Release number of selected component (if applicable):
CNV 4.10.0

How reproducible:
100%

Steps to Reproduce:
1. Create HPP and use a path that shares FS with OS

Actual results:
No feedback whatsoever

Expected results:
Feedback in the form of alert/event that path shouldn't be used

Additional info:
As alerts infrastructure is in place, it would be a good idea
to have an alert for this case, along with a runbook that explains how to fix this.

Comment 1 Jenia Peimer 2022-02-09 13:39:29 UTC
Created attachment 1860101 [details]
HPP Alert screenshot

Comment 2 Jenia Peimer 2022-02-09 13:43:20 UTC
Created attachment 1860103 [details]
HPP Alert screenshot 2

Verified on CNV v4.10.0-651, hostpath-provisioner-operator v4.10.0-61
Screenshots of the UI are attached.
 
To see the alerting logs using CLI:

Install HPP CR that shares a path with OS:

$ cat hpp-cr-basic.yaml 
apiVersion: hostpathprovisioner.kubevirt.io/v1beta1
kind: HostPathProvisioner
metadata:
 name: hostpath-provisioner
spec:
 imagePullPolicy: IfNotPresent
 storagePools:
   - name: hpp-csi-local
     path: "/var/hpvolumes"
 workload:
   nodeSelector:
     kubernetes.io/os: linux

Connect to the node:
$ oc debug node/c01-jp410-fr-pfmr4-worker-0-k6mzs
sh-4.4# chroot /host
sh-4.4# 

See where the OS is located:
sh-4.4# findmnt -T /
TARGET SOURCE                                                                                                    FSTYPE OPTIONS
/      /dev/vda4[/ostree/deploy/rhcos/deploy/11aa5111b525d060f5b3d4badc43c7acbe9bc8d6ff2949c90f9a35287c018821.0] xfs    rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsiz

See that hpp path /var/hpvolumes shares the same path as OS:
sh-4.4# findmnt -T /var/hpvolumes
TARGET SOURCE                              FSTYPE OPTIONS
/var   /dev/vda4[/ostree/deploy/rhcos/var] xfs    rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,prjquota

$ oc get pods -A | grep hostpath
openshift-cnv                                      hostpath-provisioner-csi-7m8kb                                    4/4     Running     0               9m35s
openshift-cnv                                      hostpath-provisioner-csi-hqp4b                                    4/4     Running     0               9m35s
openshift-cnv                                      hostpath-provisioner-csi-q7q2j                                    4/4     Running     0               9m35s
openshift-cnv                                      hostpath-provisioner-operator-5cc9cf795d-szctm                    1/1     Running     0               46h

$ oc logs -n openshift-cnv hostpath-provisioner-csi-7m8kb -c hostpath-provisioner
I0209 12:06:55.675874       1 plugin.go:44] Starting Prometheus metrics endpoint server
I0209 12:06:55.676247       1 plugin.go:47] Starting new HostPathDriver, config: {kubevirt.io.hostpath-provisioner unix:///csi/csi.sock c01-jp410-fr-pfmr4-worker-0-s4xcc map[]  v4.10.0 <nil>}
I0209 12:06:55.728163       1 hostpath.go:88] name: hpp-csi-local, dataDir: hpp-csi-local-data-dir/csi
I0209 12:06:55.728218       1 hostpath.go:107] Driver: kubevirt.io.hostpath-provisioner, version: v4.10.0 
I0209 12:06:55.728311       1 server.go:194] Starting domain socket: unix///csi/csi.sock
I0209 12:06:55.728416       1 server.go:89] Listening for connections on address: &net.UnixAddr{Name:"//csi/csi.sock", Net:"unix"}
I0209 12:06:55.737740       1 utils.go:221] pool (hpp-csi-local, hpp-csi-local-data-dir/csi), shares path with OS which can lead to node disk pressure
I0209 12:07:55.746494       1 utils.go:221] pool (hpp-csi-local, hpp-csi-local-data-dir/csi), shares path with OS which can lead to node disk pressure
I0209 12:08:55.764060       1 utils.go:221] pool (hpp-csi-local, hpp-csi-local-data-dir/csi), shares path with OS which can lead to node disk pressure
I0209 12:09:55.777407       1 utils.go:221] pool (hpp-csi-local, hpp-csi-local-data-dir/csi), shares path with OS which can lead to node disk pressure
I0209 12:10:55.785761       1 utils.go:221] pool (hpp-csi-local, hpp-csi-local-data-dir/csi), shares path with OS which can lead to node disk pressure
I0209 12:11:55.802204       1 utils.go:221] pool (hpp-csi-local, hpp-csi-local-data-dir/csi), shares path with OS which can lead to node disk pressure
I0209 12:12:55.814555       1 utils.go:221] pool (hpp-csi-local, hpp-csi-local-data-dir/csi), shares path with OS which can lead to node disk pressure
I0209 12:13:55.821748       1 utils.go:221] pool (hpp-csi-local, hpp-csi-local-data-dir/csi), shares path with OS which can lead to node disk pressure
I0209 12:14:55.831517       1 utils.go:221] pool (hpp-csi-local, hpp-csi-local-data-dir/csi), shares path with OS which can lead to node disk pressure

Comment 8 errata-xmlrpc 2022-03-16 16:06:09 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Virtualization 4.10.0 Images security and bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:0947


Note You need to log in before you can comment on or make changes to this bug.