Bug 2028775

Description of problem:
We're using a node-exporter container in Ceph deployments for monitoring purposes.
The issue can occur if we would like to allow node-exporter instance to collect statistics from the storage devices which are local to the host, which results in errors due to the fact that the exporter process is caged inside a container:

~~~
# curl -s localhost:9100/metrics| grep node_filesystem_device_error |  grep -v ^#
node_filesystem_device_error{device="/dev/mapper/rootvg-cephconlv",fstype="xfs",mountpoint="/var/lib/containers"} 1
node_filesystem_device_error{device="/dev/mapper/rootvg-cephconlv",fstype="xfs",mountpoint="/var/lib/containers/storage/overlay"} 1
node_filesystem_device_error{device="/dev/mapper/rootvg-cephloglv",fstype="xfs",mountpoint="/var/log/ceph"} 1
node_filesystem_device_error{device="/dev/mapper/rootvg-cephmonlv",fstype="xfs",mountpoint="/var/lib/ceph/mon"} 1
~~~

If we bind-mount (ro) host's root to the container and point that location as "--path.rootfs" option to the node-exporter running inside it, the stats are being collected as expected.

Is there any chance that we get this hard-coded somehow in the container config as we ship it? Or does presenting of the rootfs to the container reveal any security-related issues that's i'm probably not aware of?


Version-Release number of selected component (if applicable):
Reproduced on v4.6 container tag:

~~~
# podman images
REPOSITORY                                                                    TAG               IMAGE ID      CREATED       SIZE
registry.redhat.io/openshift4/ose-prometheus-node-exporter                    v4.6              486b6b9a1c3a  4 weeks ago   252 MB
~~~


How reproducible:
Always

Steps to Reproduce:
1. With default setup node-exporter on the Ceph monitors is not able to export filesystem stats for the logical volumes which are local to the host.
2. In order for node-exporter to scrub the host's local filesystem stats, its container has to be started with two modifications:
   a. Bind-mount host's / into container /rootfs (read-only).
   b. Pointing container's /rootfs as an argument for --path.rootfs for node-exporter.
3. We tested that with modifying the ExecStart commandline for the node-exporter systemd unit override:
   
   ~~~
   $ cat /etc/systemd/system/node_exporter.service.d/override.conf
   [Service]
   ExecStart=
   ExecStart=/usr/bin/podman run --rm --name=node-exporter \
     -d --log-driver journald --conmon-pidfile /%t/%n-pid --cidfile /%t/%n-cid \
     --pids-limit=0 \
     --privileged \
     -v /proc:/host/proc:ro -v /sys:/host/sys:ro -v /:/rootfs:ro \         <---
     --net=host \
     registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.1 \
     --path.procfs=/host/proc \
     --path.sysfs=/host/sys \
     --path.rootfs=/rootfs \                                               <---
     --no-collector.timex \
     
     $ sudo systemctl daemon-reload
     $ sudo systemctl restart node_exporter
     ~~~
 
4. Verified that the changes were picked up by the newly-spawned container:
   
   ~~~
   $ podman ps --no-trunc
   <...>  registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.1  --path.procfs=/host/proc --path.sysfs=/host/sys --path.rootfs=/rootfs --no-collector.timex
   --web.listen-address=:9100  18 seconds ago  Up 19 seconds ago           node-exporter
    
   $ podman exec node-exporter ls /rootfs
   bin
   boot
   dev
   etc
   home
   lib
   lib64
   media
   mnt
   opt
   proc
   root
   run
   sbin
   srv
   sys
   tmp
   usr
   var
   ~~~
   
5. Once the container is running with that, we can see that there are no errors when collecting the stats from the LVM volumes which are local to the host.

Additional info: this looks similar to the issue we have described in RHBZ #1814150, but the underlying reason is obviously different one in this particular situation.