Description of problem: It appears that we can't ssh into RHCOS nodes any more. When we do, the client authenticates fine and receives the SSH banner (i.e. /etc/motd), but it also prints the error message "PTY allocation request failed on channel 0" and the session hangs. On the server side, we see "error: openpty: No such file or directory" in the sshd logs. Stracing sshd, I can see "openat(AT_FDCWD, "/dev/ptmx", O_RDWR) = -1 EACCES (Permission denied)" I can see that sshd is running as root. /dev/ptmx is a symlink to pts/ptmx, whose permissions are: ``` ls -lZ /dev/pts/ptmx c---------. 1 root root system_u:object_r:devpts_t:s0 5, 2 Apr 16 02:11 /dev/pts/ptmx" ``` There are no AVCs in /var/log/audit/audit.log, yet, if I run `setenforce 0`, I can then ssh in, and when I run `setenforce 1` again, I can't. Version-Release number of selected component (if applicable): Seen on at least the following: 46.82.202102190441-0 44.82.202101041630-0
RHCOS is RHEL, so we are shipping the same `sshd` and SELinux policy that RHEL 8 is using. We've only made the following changes to the sshd config - https://github.com/openshift/os/blob/master/manifest.yaml#L70-L75 But they don't seem like the would cause this kind of problem. I don't think we have seen any other reports of problems SSH'ing to hosts on 4.6 or beyond. I'm not able to reproduce this with a single RHCOS 4.8 node or a single RHCOS 4.6 node. Which architecture is this on? What platform is being used? Is this repeatable across multiple nodes? Have any customizations been landed via MachineConfigs? Does this happen immediately after install or does it happen after a certain amount of time? I would see if this is repeatable on a single RHCOS node in your environment. Additionally, it would be an interesting experiment to see if this is repeatable using a RHEL 8.2 EUS host in the same environment. Please capture the complete journal from the affected node and share it with us. Additionally, try to run `sudo ausearch -m avc` to see if anything comes up. A `sosreport` might be be useful, too.
The issue we're seeing is being triggered by a recent ARO change and almost certainly isn't an RHCOS fault. For future reference, I think that if a /dev/ptmx -> pts/ptmx symlink is ever seen on the container host filesystem, it means there's a problem. ARO has introduced a new daemonset, https://github.com/Azure/ARO-RP/blob/e86474f6a8878af745ac1e0c9dea7e9bb94198a7/pkg/operator/controllers/routefix/routefix.go#L140-L268 , and that daemonset is somehow causing runc to symlink /dev/ptmx -> pts/ptmx. I don't understand why this is being triggered. I think https://github.com/projectatomic/runc/blob/7116f03b5bba4bf70e86fcd0674c18b8e6459c1e/libcontainer/rootfs_linux.go#L741-L750 is the core code in runc. For reference, https://github.com/opencontainers/runc/issues/1866 and https://github.com/opencontainers/runc/issues/80 are cases where this functionality in runc has previously caused problems. In this case, the impact is as follows: SELinux does not allow sshd to open /dev/ptmx when it is a symlink, and this prevents anyone from being able to ssh onto the node. Using the following auditd rules, I have been able to prove that it is runc that is creating the symlink: ``` # tail -2 /etc/audit/rules.d/audit.rules -a always,exit -F arch=b64 -S symlinkat -F key=symlinkat_rule -a always,exit -F arch=b64 -S symlink -F key=symlink_rule ``` Here is an example of the problem taking place. Note comm="runc:[2:INIT]": ``` # grep 1618610224.089:877 /var/log/audit/audit.log type=SYSCALL msg=audit(1618610224.089:877): arch=c000003e syscall=266 success=yes exit=0 a0=c0000dba05 a1=ffffffffffffff9c a2=c00029b780 a3=c00029b780 items=3 ppid=4794 pid=4858 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="runc:[2:INIT]" exe="/" subj=system_u:system_r:container_runtime_t:s0 key="symlinkat_rule"ARCH=x86_64 SYSCALL=symlinkat AUID="unset" UID="root" GID="root" EUID="root" SUID="root" FSUID="root" EGID="root" SGID="root" FSGID="root" type=CWD msg=audit(1618610224.089:877): cwd="/var/lib/containers/storage/overlay/fa1152e5b9cf192cf3966a232a97617f9335b0e3d74aed4767cd7e07cee02e18/merged" type=PATH msg=audit(1618610224.089:877): item=0 name="pts/ptmx" nametype=UNKNOWN cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0 type=PATH msg=audit(1618610224.089:877): item=1 name="/var/lib/containers/storage/overlay/fa1152e5b9cf192cf3966a232a97617f9335b0e3d74aed4767cd7e07cee02e18/merged/dev/" inode=1025 dev=00:06 mode=040755 ouid=0 ogid=0 rdev=00:00 obj=system_u:object_r:device_t:s0 nametype=PARENT cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0OUID="root" OGID="root" type=PATH msg=audit(1618610224.089:877): item=2 name="/var/lib/containers/storage/overlay/fa1152e5b9cf192cf3966a232a97617f9335b0e3d74aed4767cd7e07cee02e18/merged/dev/ptmx" inode=59635 dev=00:06 mode=0120777 ouid=0 ogid=0 rdev=00:00 obj=system_u:object_r:device_t:s0 nametype=CREATE cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0OUID="root" OGID="root" type=PROCTITLE msg=audit(1618610224.089:877): proctitle=2F7573722F62696E2F72756E6300696E6974 ``` And here is the symlink itself. Note the ctime: 2021-04-16 21:57:04.089337077 +0000 == 1618610224.089 in the audit logs. ``` # ls -l /dev/ptmx lrwxrwxrwx. 1 root root 8 Apr 16 21:57 /dev/ptmx -> pts/ptmx # stat /dev/ptmx File: /dev/ptmx -> pts/ptmx Size: 8 Blocks: 0 IO Block: 4096 symbolic link Device: 6h/6d Inode: 59635 Links: 1 Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) Context: system_u:object_r:device_t:s0 Access: 2021-04-16 21:57:38.899469613 +0000 Modify: 2021-04-16 21:57:04.089337077 +0000 Change: 2021-04-16 21:57:04.089337077 +0000 Birth: - ``` Finally, here's an example of the AVC (when temporarily running with `setenforce 0`): ``` type=AVC msg=audit(1618584132.433:121145): avc: denied { open } for pid=3538293 comm="sshd" path="/dev/pts/ptmx" dev="devpts" ino=2 scontext=system_u:system_r:sshd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:devpts_t:s0 tclass=chr_file permissive=1 ``` 1. Is there an underlying runc issue here? How have we managed to trigger this codepath and is there a risk that others could too? 2. ARO team needs help to understand how we can avoid triggering this codepath and update our daemonset accordingly. We will then need to rollout and clear the condition across all of our clusters; probably a rolling reboot is the best way to do this.
Example runc RPM version is runc-1.0.0-82.rhaos4.6.git086e841.el8.x86_64 .
Dan Walsh, Thoughts on who or which group should look at this?
captured verbose connection through ssh # ssh -vvv -i azure core.0.9 OpenSSH_8.0p1, OpenSSL 1.1.1c FIPS 28 May 2019 debug1: Reading configuration data /etc/ssh/ssh_config debug3: /etc/ssh/ssh_config line 51: Including file /etc/ssh/ssh_config.d/05-redhat.conf depth 0 debug1: Reading configuration data /etc/ssh/ssh_config.d/05-redhat.conf debug2: checking match for 'final all' host 10.0.0.9 originally 10.0.0.9 debug3: /etc/ssh/ssh_config.d/05-redhat.conf line 3: not matched 'final' debug2: match not found debug3: /etc/ssh/ssh_config.d/05-redhat.conf line 5: Including file /etc/crypto-policies/back-ends/openssh.config depth 1 (parse only) debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config debug3: gss kex names ok: [gss-gex-sha1-,gss-group14-sha1-] debug3: kex names ok: [curve25519-sha256,curve25519-sha256,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha 256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1] debug1: configuration requests final Match pass debug2: resolve_canonicalize: hostname 10.0.0.9 is address debug1: re-parsing configuration debug1: Reading configuration data /etc/ssh/ssh_config debug3: /etc/ssh/ssh_config line 51: Including file /etc/ssh/ssh_config.d/05-redhat.conf depth 0 debug1: Reading configuration data /etc/ssh/ssh_config.d/05-redhat.conf debug2: checking match for 'final all' host 10.0.0.9 originally 10.0.0.9 debug3: /etc/ssh/ssh_config.d/05-redhat.conf line 3: matched 'final' debug2: match found debug3: /etc/ssh/ssh_config.d/05-redhat.conf line 5: Including file /etc/crypto-policies/back-ends/openssh.config depth 1 debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config debug3: gss kex names ok: [gss-gex-sha1-,gss-group14-sha1-] debug3: kex names ok: [curve25519-sha256,curve25519-sha256,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha 256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1] debug2: ssh_connect_direct debug1: Connecting to 10.0.0.9 [10.0.0.9] port 22. debug1: Connection established. debug1: identity file azure type -1 debug1: identity file azure-cert type -1 debug1: Local version string SSH-2.0-OpenSSH_8.0 debug1: Remote protocol version 2.0, remote software version OpenSSH_8.0 debug1: match: OpenSSH_8.0 pat OpenSSH* compat 0x04000000 debug2: fd 5 setting O_NONBLOCK debug1: Authenticating to 10.0.0.9:22 as 'core' debug3: hostkeys_foreach: reading file "/root/.ssh/known_hosts" debug3: record_hostkey: found key type ECDSA in file /root/.ssh/known_hosts:1 debug3: load_hostkeys: loaded 1 keys from 10.0.0.9 debug3: order_hostkeyalgs: prefer hostkeyalgs: ecdsa-sha2-nistp256-cert-v01,ecdsa-sha2-nistp384-cert-v01,ecdsa-sha2-nistp521-cert-v01,ecdsa-sha2-nistp256, ecdsa-sha2-nistp384,ecdsa-sha2-nistp521 debug3: send packet: type 20 debug1: SSH2_MSG_KEXINIT sent debug3: receive packet: type 20 debug1: SSH2_MSG_KEXINIT received debug2: local client KEXINIT proposal debug2: KEX algorithms: curve25519-sha256,curve25519-sha256,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sh a256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,ext-info-c debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01,ecdsa-sha2-nistp384-cert-v01,ecdsa-sha2-nistp521-cert-v01,ecdsa-sha2-nistp256,ecdsa-sha2-nistp38 4,ecdsa-sha2-nistp521,ssh-ed25519-cert-v01,rsa-sha2-512-cert-v01,rsa-sha2-256-cert-v01,ssh-rsa-cert-v01,ssh-ed25519,rsa-sha2-512,rsa-sha2-256, ssh-rsa debug2: ciphers ctos: aes256-gcm,chacha20-poly1305,aes256-ctr,aes256-cbc,aes128-gcm,aes128-ctr,aes128-cbc debug2: ciphers stoc: aes256-gcm,chacha20-poly1305,aes256-ctr,aes256-cbc,aes128-gcm,aes128-ctr,aes128-cbc debug2: MACs ctos: hmac-sha2-256-etm,hmac-sha1-etm,umac-128-etm,hmac-sha2-512-etm,hmac-sha2-256,hmac-sha1,umac-128,hmac-sha2-512 debug2: MACs stoc: hmac-sha2-256-etm,hmac-sha1-etm,umac-128-etm,hmac-sha2-512-etm,hmac-sha2-256,hmac-sha1,umac-128,hmac-sha2-512 debug2: compression ctos: none,zlib,zlib debug2: compression stoc: none,zlib,zlib debug2: languages ctos: debug2: languages stoc: debug2: first_kex_follows 0 debug2: reserved 0 debug2: peer server KEXINIT proposal debug2: KEX algorithms: curve25519-sha256,curve25519-sha256,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sh a256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1 debug2: host key algorithms: rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256,ssh-ed25519 debug2: ciphers ctos: aes256-gcm,chacha20-poly1305,aes256-ctr,aes256-cbc,aes128-gcm,aes128-ctr,aes128-cbc debug2: ciphers stoc: aes256-gcm,chacha20-poly1305,aes256-ctr,aes256-cbc,aes128-gcm,aes128-ctr,aes128-cbc debug2: MACs ctos: hmac-sha2-256-etm,hmac-sha1-etm,umac-128-etm,hmac-sha2-512-etm,hmac-sha2-256,hmac-sha1,umac-128,hmac-sha2-512 debug2: MACs stoc: hmac-sha2-256-etm,hmac-sha1-etm,umac-128-etm,hmac-sha2-512-etm,hmac-sha2-256,hmac-sha1,umac-128,hmac-sha2-512 debug2: compression ctos: none,zlib debug2: compression stoc: none,zlib debug2: languages ctos: debug2: languages stoc: debug2: first_kex_follows 0 debug2: reserved 0 debug1: kex: algorithm: curve25519-sha256 debug1: kex: host key algorithm: ecdsa-sha2-nistp256 debug1: kex: server->client cipher: aes256-gcm MAC: <implicit> compression: none debug1: kex: client->server cipher: aes256-gcm MAC: <implicit> compression: none debug1: kex: curve25519-sha256 need=32 dh_need=32 debug1: kex: curve25519-sha256 need=32 dh_need=32 debug3: send packet: type 30 debug1: expecting SSH2_MSG_KEX_ECDH_REPLY debug3: receive packet: type 31 debug1: Server host key: ecdsa-sha2-nistp256 SHA256:eyArPGWd+EE2vdRbqLliSYnVeIOBR14d9qTeldwU5yo debug3: hostkeys_foreach: reading file "/root/.ssh/known_hosts" debug3: record_hostkey: found key type ECDSA in file /root/.ssh/known_hosts:1 debug3: load_hostkeys: loaded 1 keys from 10.0.0.9 debug1: Host '10.0.0.9' is known and matches the ECDSA host key. debug1: Found key in /root/.ssh/known_hosts:1 debug3: send packet: type 21 debug2: set_newkeys: mode 1 debug1: rekey out after 4294967296 blocks debug1: SSH2_MSG_NEWKEYS sent debug1: expecting SSH2_MSG_NEWKEYS debug3: receive packet: type 21 debug1: SSH2_MSG_NEWKEYS received debug2: set_newkeys: mode 0 debug1: rekey in after 4294967296 blocks debug1: Will attempt key: azure explicit debug2: pubkey_prepare: done debug3: send packet: type 5 debug3: receive packet: type 7 debug1: SSH2_MSG_EXT_INFO received debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521> debug3: receive packet: type 6 debug2: service_accept: ssh-userauth debug1: SSH2_MSG_SERVICE_ACCEPT received debug3: send packet: type 50 debug3: receive packet: type 51 debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic debug3: start over, passed a different list publickey,gssapi-keyex,gssapi-with-mic debug3: preferred gssapi-with-mic,publickey,keyboard-interactive,password debug3: authmethod_lookup gssapi-with-mic debug3: remaining preferred: publickey,keyboard-interactive,password debug3: authmethod_is_enabled gssapi-with-mic debug1: Next authentication method: gssapi-with-mic debug1: Unspecified GSS failure. Minor code may provide more information No Kerberos credentials available (default cache: KEYRING:persistent:0) debug2: we did not send a packet, disable method debug3: authmethod_lookup publickey debug3: remaining preferred: keyboard-interactive,password debug3: authmethod_is_enabled publickey debug1: Next authentication method: publickey debug1: Trying private key: azure debug3: sign_and_send_pubkey: RSA SHA256:IzxSBt81ShsI6g/x0WJVPQUY+Gv6KTUVm/zafuSX/KU debug3: sign_and_send_pubkey: signing using rsa-sha2-256 debug3: send packet: type 50 debug2: we sent a publickey packet, wait for reply debug3: receive packet: type 52 debug1: Authentication succeeded (publickey). Authenticated to 10.0.0.9 ([10.0.0.9]:22). debug1: channel 0: new [client-session] debug3: ssh_session2_open: channel_new: 0 debug2: channel 0: send open debug3: send packet: type 90 debug1: Requesting no-more-sessions debug3: send packet: type 80 debug1: Entering interactive session. debug1: pledge: network debug3: receive packet: type 80 debug1: client_input_global_request: rtype hostkeys-00 want_reply 0 debug3: receive packet: type 4 debug1: Remote: /var/home/core/.ssh/authorized_keys:2: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding debug3: receive packet: type 91 debug2: channel_input_open_confirmation: channel 0: callback start debug2: fd 5 setting TCP_NODELAY debug3: ssh_packet_set_tos: set IP_TOS 0x48 debug2: client_session2_setup: id 0 debug2: channel 0: request pty-req confirm 1 debug3: send packet: type 98 debug1: Sending environment. debug3: Ignored env LS_COLORS debug3: Ignored env OS_GIT_MINOR debug3: Ignored env SOURCE_GIT_URL debug3: Ignored env HOSTNAME debug3: Ignored env OLDPWD debug3: Ignored env OS_GIT_TREE_STATE debug3: Ignored env BUILD_VERSION debug3: Ignored env OS_GIT_COMMIT debug3: Ignored env KUBERNETES_PORT_443_TCP_PROTO debug3: Ignored env KUBERNETES_PORT_443_TCP_ADDR debug3: Ignored env container debug3: Ignored env KUBERNETES_PORT debug3: Ignored env PWD debug3: Ignored env HOME debug3: Ignored env OS_GIT_VERSION debug3: Ignored env OS_GIT_MAJOR debug3: Ignored env KUBERNETES_SERVICE_PORT_HTTPS debug3: Ignored env KUBERNETES_PORT_443_TCP_PORT debug3: Ignored env SOURCE_GIT_COMMIT debug3: Ignored env KUBERNETES_PORT_443_TCP debug3: Ignored env TERM debug3: Ignored env NSS_SDB_USE_CACHE debug3: Ignored env GODEBUG debug3: Ignored env SHLVL debug3: Ignored env KUBERNETES_SERVICE_PORT debug3: Ignored env SOURCE_DATE_EPOCH debug3: Ignored env OS_GIT_PATCH debug3: Ignored env SOURCE_GIT_TREE_STATE debug3: Ignored env SOURCE_GIT_TAG debug3: Ignored env __doozer debug3: Ignored env BUILD_RELEASE debug3: Ignored env PATH debug3: Ignored env KUBERNETES_SERVICE_HOST debug3: Ignored env LESSOPEN debug3: Ignored env _ debug2: channel 0: request shell confirm 1 debug3: send packet: type 98 debug2: channel_input_open_confirmation: channel 0: callback done debug2: channel 0: open confirm rwindow 0 rmax 32768 debug3: receive packet: type 100 debug2: channel_input_status_confirm: type 100 id 0 debug2: channel_input_status_confirm: type 100 id 0 PTY allocation request failed on channel 0 debug2: channel 0: rcvd adjust 2097152 debug3: receive packet: type 99 debug2: channel_input_status_confirm: type 99 id 0 debug2: shell request accepted on channel 0 Red Hat Enterprise Linux CoreOS 46.82.202103050041-0 Part of OpenShift 4.6, RHCOS is a Kubernetes native operating system managed by the Machine Config Operator (`clusteroperator/machine-config`). WARNING: Direct SSH access to machines is not recommended; instead, make configuration changes via `machineconfig` objects: https://docs.openshift.com/container-platform/4.6/architecture/architecture-rhcos.html ---
@
The question is what is causing the symlink? Rather then separate devices. Even if we fixed and allowed the SELinux failure. the permissions on /dev/pts/ptmx Would prevent most non running processes from using the device.
Are you running any special privileged containers? My best guess is one of them is doing something like `-v /dev:/dev` which I believe can cause these symptoms. Don't use `-v /dev:/dev`.
The problem starts (ssh login no longer possible) once we deploy the following https://access.redhat.com/solutions/5252831
if you use -v /dev:/dev, you can add `--mount type=devpts,destination=/dev/pts` so a new instance of devpts is mounted inside the container
Yes I think we should at least WARN on this, if not totally deny it. What do Colin and Giuseppe think?
I think a WARN won't be enough as it is already too late and the container can damage the host setup. IMO, such check should not be in the OCI runtime as it could deny other valid use cases, but we can play safe and add it to CRI-O.
Interesting part here is that in this case it was mounted as readonly filesytem, but this mapping / -> / caused device to be mapped. So in theory there can be usecase where I want to mount everything and mount my script from ConfigMap and do something (feasible but crazy). So denying might be too hard. On the other hand notification will never reach OpenShift users as I dont think there is way to propagate this to the UI/CLI via kube API? So what is left?
Most likely in those cases, you should selectively mount the host into the container -v /usr:/usr -v /run:/run -v /etc:/etc ... The problem with /:/ is you end up with /proc:/proc /dev:/dev /sys:/sys which can cause complications and unexpected results.
I would like to experiment with something like podman run --rootfs / ... Where Podman was smart enough to still mount over the dangerous parts like /dev, /sys, /proc And maybe may over /run, Mount /var as an overlay. Would be a nice experiment to see if you could easily containerize the host system without causing damage to it. Most likely this would require --security-opt label=disable, since SELinux would block a lot of access.
I'm going to assign this to Giuseppe to chase down the warning or error part of this.
opened a PR for CRI-O: https://github.com/cri-o/cri-o/pull/4853 I've changed idea about having it a hard error. While it is unlikely, it could block valid use cases we are not aware of
There are a couple of pieces of work going on: - CRI-O PR, which warns a user about it: https://github.com/cri-o/cri-o/pull/4853 - upstream k8s PR to prevent a pod from doing this entirely: https://github.com/kubernetes/kubernetes/issues/101749 but ultimately, both of those are the two tools automating the process of educating users about how dangerous mounting "/" to "/" is. They'll both help, but we should first address the lack of documentation saying how it is not a good idea. As such, moving to documentation to begin the process of education.
Peter -- Is the request here to document that the user should not mount to `'`? Is this true of any mount? volumes.hostPath: containers.volumeMounts.mountPath: volumes.nfs.path Anywhere else? Michael
Specifically the ask is to not mount "/" to "/". Basically, the user should be wary of mounting any path on the host to any same path in the container where the path in the container may have changes that shouldn't be added to the host. This specific bug manifests because /dev/ptmx must be edited by the runtime for the container, but that modification shouldn't be applied to the host. Any hostPath volume should be affected. I am not sure about nfs but I don't think users will be as tempted to mount '/' from an nfs volume anyway. I'd say it's worth adding, just in case.
Peter -- Can you please take a look at: https://github.com/openshift/openshift-docs/pull/34464 Thank you, Michael
Xiaoli -- Can you please assign this BZ and PR for QE review? Thank you in advance. https://github.com/openshift/openshift-docs/pull/34464 Michael
Changes are live: https://docs.openshift.com/container-platform/4.8/nodes/containers/nodes-containers-volumes.html#nodes-containers-volumes-adding_nodes-containers-volumes