Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1917504

Summary:

Security patch for CVE-2020-25212 breaks directory listings via 'ls' on NFS V4.2 shares mounted with selinux enabled labels

Product:

Red Hat Enterprise Linux 7

Reporter:

Dave Wysochanski <dwysocha>

Component:

kernel

Assignee:

Dave Wysochanski <dwysocha>

kernel sub component:

NFS

QA Contact:

Murphy Zhou <xzhou>

Status:

CLOSED ERRATA

Docs Contact:

Severity:

urgent

Priority:

urgent

CC:

ajb, bfields, bhu, dhoward, fsorenso, nfs-maint, nmurray, phil, riehecky, tom, toracat, xzhou, yoyang

Version:

7.9

Keywords:

Regression, Reproducer, ZStream

Target Milestone:

Target Release:

---

Hardware:

Unspecified

OS:

Unspecified

Whiteboard:

Fixed In Version:

kernel-3.10.0-1160.15.2.el7

Doc Type:

If docs needed, set a value

Doc Text:

Story Points:

---

Clone Of:

Clones:

1919141 1919144 1919145 (view as bug list)

Environment:

Last Closed:

2021-02-02 12:02:36 UTC

Type:

Bug

Regression:

---

Mount Type:

---

Documentation:

---

CRM:

Verified Versions:

Category:

---

oVirt Team:

---

RHEL 7.3 requirements from Atomic Host:

Cloudforms Team:

---

Target Upstream Version:

Embargoed:

Bug Depends On:

Bug Blocks:

1904388, 1919141, 1919144, 1919145

Attachments:

Description	Flags
v1 of reproducer for this bug - could use a bit of cleanup but works to show the problem	none
v2 of reproducer for this bug	none

Description Dave Wysochanski 2021-01-18 15:40:23 UTC

Description of problem:
From customer:
On any of our diskless systems that are running RHEL 7.9, with kernel "kernel-3.10.0-1160.11.1.el7.x86_64", Any share mounted via NFS V4.2 is broken. Our diskless nodes have SE Linux enabled, and are exporting SE Linux permissions as part of these NFS shares, which is why we need V4.2. When we run an "ls" command on a mounted NFS share, there are no results, so things like shell command tab completion do not work. However, if we run an ls command and specify a direct file on a mounted NFS share, it does show.

If we mount our NFS shares with an older NFS version, we are able to see our files. However, older NFS versions do not support SE Linux, therefore this is not a solution for us.

We believe that the change in the kernel for CVE-2020-25212 is the culprit here.
We reverted back to kernel-3.1.0-1160.6.1 and do not see this issue occur. We investigated further, and rolled back the changes to the two files listed in this CVE, and recompiled kernel-3.10.0-1160.11.1, and no longer see the issue.

Version-Release number of selected component (if applicable):
RHEL 7.9 with kernel-3.10.0-1160.11.1.el7.x86_64
NFS4.2 with selinux / labelled NFS enabled

How reproducible:
Everytime

Steps to Reproduce:
TBD bug see customer description above

Actual results:
directory listings no longer work

Expected results:
directory listings work

Additional info:
When does the behavior occur? Frequency? Repeatedly? At certain times?
This issue occurs with any NFS V4.2 share on kernel-3.10.0-1160.11.1.el7.x86_64.

What information can you provide around timeframes and the business impact?
Reverting the changes to CVE-2020-25212 resolves our issues, but in theory reopens our systems to exposure to CVE-2020-25212.

Comment 4 Dave Wysochanski 2021-01-18 15:46:39 UTC

Probably we need the below patch.  Suggest creating a test kernel with customer's version plus this patch and see if the problem is resolved.  
If it is this is probably going to be a mess since we have other RHEL7 streams with this CVE patch (see https://bugzilla.redhat.com/show_bug.cgi?id=1877575#c20)

commit d33030e2ee3508d65db5644551435310df86010e
Author: Jeffrey Mitchell <jeffrey.mitchell>
Date:   Tue Sep 15 16:42:52 2020 -0500

    nfs: Fix security label length not being reset
    
    nfs_readdir_page_filler() iterates over entries in a directory, reusing
    the same security label buffer, but does not reset the buffer's length.
    This causes decode_attr_security_label() to return -ERANGE if an entry's
    security label is longer than the previous one's. This error, in
    nfs4_decode_dirent(), only gets passed up as -EAGAIN, which causes another
    failed attempt to copy into the buffer. The second error is ignored and
    the remaining entries do not show up in ls, specifically the getdents64()
    syscall.
    
    Reproduce by creating multiple files in NFS and giving one of the later
    files a longer security label. ls will not see that file nor any that are
    added afterwards, though they will exist on the backend.
    
    In nfs_readdir_page_filler(), reset security label buffer length before
    every reuse
    
    Signed-off-by: Jeffrey Mitchell <jeffrey.mitchell>
    Fixes: b4487b935452 ("nfs: Fix getxattr kernel panic and memory overflow")
    Signed-off-by: Trond Myklebust <trond.myklebust>

diff --git a/fs/nfs/dir.c b/fs/nfs/dir.c
index e732580fe47b..cb52db9a0cfb 100644
--- a/fs/nfs/dir.c
+++ b/fs/nfs/dir.c
@@ -579,6 +579,9 @@ int nfs_readdir_page_filler(nfs_readdir_descriptor_t *desc, struct nfs_entry *en
        xdr_set_scratch_buffer(&stream, page_address(scratch), PAGE_SIZE);
 
        do {
+               if (entry->label)
+                       entry->label->len = NFS4_MAXLABELLEN;
+
                status = xdr_decode(desc, entry, &stream);
                if (status != 0) {
                        if (status == -EAGAIN)

Comment 6 Frank Sorenson 2021-01-18 17:24:33 UTC

create 2 files, with the second having a longer-length label:

# echo file 1 > file_1 ; chcon -t public_context_1 file_1
# echo file 2 > file_2 ; chcon -t public_context__2 file_2

# ls -lZ
-rw-r--r--. root root unconfined_u:object_r:public_context_1:s0 file_1
-rw-r--r--. root root unconfined_u:object_r:public_context__2:s0 file_2

and over nfs:

# ls -l
total 4
-rw-r--r--. 1 root root 7 Jan 18 11:22 file_1


only the first file is shown in the directory listing, however both can be accessed:


# cat file_1
file 1

# cat file_2
file 2

Comment 7 Dave Wysochanski 2021-01-18 18:40:27 UTC

Created attachment 1748508 [details]
v1 of reproducer for this bug - could use a bit of cleanup but works to show the problem

Reproducer for this bug.  Note you only need:
- mount with "vers=4.2" and an export with "security_label" option
- 2 files on the export, the second one with a label longer than the first

[root@rhel7u9-node1 1917504]# ./t0_bz1917504.sh 
Assuming NFS vers=4.2 and filesystem xfs and mount options rw
Mon Jan 18 13:24:40 EST 2021: 0. On NFS server, export one subdirectory with specific filesystem and mount opts
/dev/loop0 /export/bz1917504-export xfs rw,seclabel,relatime,attr2,inode64,noquota 0 0
Mon Jan 18 13:24:40 EST 2021: 1. On NFS server, create two files in the export, with different size labels
Mon Jan 18 13:24:40 EST 2021: Running command: echo 1 > /export/bz1917504-export/file1
Mon Jan 18 13:24:40 EST 2021: Running command: echo 2 > /export/bz1917504-export/file2
Mon Jan 18 13:24:40 EST 2021: Running command: semanage fcontext -a -t public_content_t /export/bz1917504-export/file2
Mon Jan 18 13:24:45 EST 2021: Running command: restorecon -F -R -v /export/bz1917504-export
restorecon reset /export/bz1917504-export context system_u:object_r:unlabeled_t:s0->system_u:object_r:usr_t:s0
restorecon reset /export/bz1917504-export/file1 context unconfined_u:object_r:unlabeled_t:s0->system_u:object_r:usr_t:s0
restorecon reset /export/bz1917504-export/file2 context unconfined_u:object_r:unlabeled_t:s0->system_u:object_r:public_content_t:s0
Mon Jan 18 13:24:45 EST 2021: 2. On NFS client, mount -o vers=4.2, 127.0.0.1:/export/bz1917504-export /mnt/bz1917504-mnt
Mon Jan 18 13:24:45 EST 2021: mount -o vers=4.2 127.0.0.1:/export/bz1917504-export /mnt/bz1917504-mnt
Mon Jan 18 13:24:47 EST 2021: 3. On NFS client, list the directory and see if both files show up
Mon Jan 18 13:24:47 EST 2021: directory listing showed 1 != 2
FAILED TEST ./t0_bz1917504.sh on kernel 3.10.0-1160.11.1.el7.x86_64 with NFS vers=4.2
[root@rhel7u9-node1 1917504]#


REBOOT to older kernel:

[root@rhel7u9-node1 1917504]# ./t0_bz1917504.sh 
Assuming NFS vers=4.2 and filesystem xfs and mount options rw
Trying to start nfs-server
Mon Jan 18 13:30:01 EST 2021: 0. On NFS server, export one subdirectory with specific filesystem and mount opts
/dev/loop0 /export/bz1917504-export xfs rw,seclabel,relatime,attr2,inode64,noquota 0 0
Mon Jan 18 13:30:01 EST 2021: 1. On NFS server, create two files in the export, with different size labels
Mon Jan 18 13:30:01 EST 2021: Running command: echo 1 > /export/bz1917504-export/file1
Mon Jan 18 13:30:01 EST 2021: Running command: echo 2 > /export/bz1917504-export/file2
Mon Jan 18 13:30:01 EST 2021: Running command: semanage fcontext -a -t public_content_t /export/bz1917504-export/file2
Mon Jan 18 13:30:07 EST 2021: Running command: restorecon -F -R -v /export/bz1917504-export
restorecon reset /export/bz1917504-export context system_u:object_r:unlabeled_t:s0->system_u:object_r:usr_t:s0
restorecon reset /export/bz1917504-export/file1 context unconfined_u:object_r:unlabeled_t:s0->system_u:object_r:usr_t:s0
restorecon reset /export/bz1917504-export/file2 context unconfined_u:object_r:unlabeled_t:s0->system_u:object_r:public_content_t:s0
Mon Jan 18 13:30:07 EST 2021: 2. On NFS client, mount -o vers=4.2, 127.0.0.1:/export/bz1917504-export /mnt/bz1917504-mnt
Mon Jan 18 13:30:07 EST 2021: mount -o vers=4.2 127.0.0.1:/export/bz1917504-export /mnt/bz1917504-mnt
Mon Jan 18 13:30:10 EST 2021: 3. On NFS client, list the directory and see if both files show up
PASSED TEST ./t0_bz1917504.sh on kernel 3.10.0-1160.6.1.el7.x86_64 with NFS vers=4.2
[root@rhel7u9-node1 1917504]#

Comment 9 Dave Wysochanski 2021-01-18 18:56:37 UTC

[root@rhel7u9-node1 1917504]# ./t0_bz1917504.sh 
Assuming NFS vers=4.2 and filesystem xfs and mount options rw
Trying to start nfs-server
Mon Jan 18 13:56:11 EST 2021: 0. On NFS server, export one subdirectory with specific filesystem and mount opts
/dev/loop0 /export/bz1917504-export xfs rw,seclabel,relatime,attr2,inode64,noquota 0 0
Mon Jan 18 13:56:12 EST 2021: 1. On NFS server, create two files in the export, with different size labels
Mon Jan 18 13:56:12 EST 2021: Running command: echo 1 > /export/bz1917504-export/file1
Mon Jan 18 13:56:12 EST 2021: Running command: echo 2 > /export/bz1917504-export/file2
Mon Jan 18 13:56:12 EST 2021: Running command: semanage fcontext -a -t public_content_t /export/bz1917504-export/file2
Mon Jan 18 13:56:17 EST 2021: Running command: restorecon -F -R -v /export/bz1917504-export
restorecon reset /export/bz1917504-export context system_u:object_r:unlabeled_t:s0->system_u:object_r:usr_t:s0
restorecon reset /export/bz1917504-export/file1 context unconfined_u:object_r:unlabeled_t:s0->system_u:object_r:usr_t:s0
restorecon reset /export/bz1917504-export/file2 context unconfined_u:object_r:unlabeled_t:s0->system_u:object_r:public_content_t:s0
Mon Jan 18 13:56:17 EST 2021: 2. On NFS client, mount -o vers=4.2, 127.0.0.1:/export/bz1917504-export /mnt/bz1917504-mnt
Mon Jan 18 13:56:17 EST 2021: mount -o vers=4.2 127.0.0.1:/export/bz1917504-export /mnt/bz1917504-mnt
Mon Jan 18 13:56:20 EST 2021: 3. On NFS client, list the directory and see if both files show up
PASSED TEST ./t0_bz1917504.sh on kernel 3.10.0-1160.11.1.el7.bz1917504.1.x86_64 with NFS vers=4.2
[root@rhel7u9-node1 1917504]#

Comment 17 Dave Wysochanski 2021-01-20 18:02:08 UTC

Created attachment 1749129 [details]
v2 of reproducer for this bug

Small tweaks to run on rhel8 as well as rhel7
- use "ls -1Z"
- always show the output of the listing
- in cleanup_exit remove the label with "semanage fcontext -d"

Comment 19 Augusto Caringi 2021-01-22 01:26:44 UTC

Patch(es) committed on kernel-3.10.0-1160.15.2.el7

Comment 27 Murphy Zhou 2021-01-23 08:45:23 UTC

*** Bug 1916129 has been marked as a duplicate of this bug. ***

Comment 28 Yongcheng Yang 2021-01-26 04:12:45 UTC

*** Bug 1912550 has been marked as a duplicate of this bug. ***

Comment 32 errata-xmlrpc 2021-02-02 12:02:36 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: kernel security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:0336