Bug 2142657

Summary: [RHEL9] fuse readdir cache sometimes corrupted
Product: Red Hat Enterprise Linux 9 Reporter: Miklos Szeredi <mszeredi>
Component: kernelAssignee: Miklos Szeredi <mszeredi>
kernel sub component: File Systems - Other QA Contact: Boyang Xue <bxue>
Status: CLOSED ERRATA Docs Contact:
Severity: unspecified    
Priority: unspecified CC: bxue, dhowells, fsorenso, hartsjc, mszeredi, swhiteho, xzhou
Version: 9.0Keywords: Triaged
Target Milestone: rcFlags: pm-rhel: mirror+
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: kernel-5.14.0-201.el9 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 2131391 Environment:
Last Closed: 2023-05-09 08:07:27 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 2131391    
Bug Blocks:    

Comment 4 Boyang Xue 2022-11-23 07:04:57 UTC
TEST PASS

Reproduced on kernel-5.14.0-197.el9 (https://beaker.engineering.redhat.com/jobs/7267697)
```
[ 1104.052836] ------------[ cut here ]------------ 
[ 1104.053454] WARNING: CPU: 3 PID: 30871 at fs/fuse/readdir.c:401 fuse_parse_cache+0x13c/0x1a0 [fuse] 
[ 1104.054367] Modules linked in: tls rfkill sunrpc vfat fat drm fuse xfs libcrc32c crct10dif_ce ghash_ce sha2_ce sha256_arm64 sha1_ce virtio_net virtio_console net_failover failover virtio_blk virtio_mmio dm_mirror dm_region_hash dm_log dm_mod 
[ 1104.056985] CPU: 3 PID: 30871 Comm: walk_tree Kdump: loaded Tainted: G        W        --------- ---  5.14.0-197.el9.aarch64 #1 
[ 1104.058061] Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015 
[ 1104.058881] pstate: 20400005 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) 
[ 1104.059787] pc : fuse_parse_cache+0x13c/0x1a0 [fuse] 
[ 1104.060238] lr : fuse_parse_cache+0x8c/0x1a0 [fuse] 
[ 1104.060867] sp : ffff80000c31bc80 
[ 1104.061237] x29: ffff80000c31bc80 x28: ffff80000c31bdd0 x27: ffff00032ff60800 
[ 1104.062154] x26: ffff00048fb77058 x25: 0000000000000000 x24: ffff00048fb77000 
[ 1104.062954] x23: 0000000000000060 x22: 000000000000001c x21: ffff00048fb77040 
[ 1104.063739] x20: 0000000000000040 x19: 0000000000000038 x18: 0000000000000000 
[ 1104.064524] x17: 0000000000000000 x16: ffffaa4af116a124 x15: 0000000000000000 
[ 1104.065310] x14: 0000000000000000 x13: 0000000000000000 x12: 0000000000000000 
[ 1104.066098] x11: 0000ffffffffffff x10: 00000000000010d0 x9 : ffffaa4a7f58e19c 
[ 1104.066881] x8 : ffff0000c6a73ab0 x7 : 000000000080c121 x6 : 000000000000002e 
[ 1104.067753] x5 : 0000000000000000 x4 : 0000000000000020 x3 : 0000000000000020 
[ 1104.068731] x2 : 000000000000001c x1 : 000000000000002f x0 : ffff00048fb77058 
[ 1104.069519] Call trace: 
[ 1104.069692]  fuse_parse_cache+0x13c/0x1a0 [fuse] 
[ 1104.070316]  fuse_readdir_cached+0x26c/0x310 [fuse] 
[ 1104.070948]  fuse_readdir+0x7c/0x130 [fuse] 
[ 1104.071244]  iterate_dir+0x164/0x1b0 
[ 1104.071796]  __arm64_sys_getdents64+0x6c/0x15c 
[ 1104.072404]  invoke_syscall.constprop.0+0x7c/0xd0 
[ 1104.072731]  el0_svc_common.constprop.0+0x154/0x160 
[ 1104.073362]  do_el0_svc+0x2c/0xb0 
[ 1104.073887]  el0_svc+0x38/0x190 
[ 1104.074108]  el0t_64_sync_handler+0x9c/0x120 
[ 1104.074697]  el0t_64_sync+0x174/0x178 
[ 1104.075240] ---[ end trace fc94e5399a1dac18 ]--- 
```

Verified on kernel-5.14.0-196.mr1633_221115_1046.el9 (https://beaker.engineering.redhat.com/jobs/7267651)
```
::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
::   Test
::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

:: [ 22:53:25 ] :: [  BEGIN   ] :: Running './walk_tree 8 /cvmfs/oasis.opensciencegrid.org'
tid 19517, child 0: alive
tid 19518, child 1: alive
tid 19519, child 2: alive
tid 19520, child 3: alive
tid 19521, child 4: alive
tid 19522, child 5: alive
tid 19523, child 6: alive
tid 19524, child 7: alive
tid 19521, child 4: exiting with NO ERROR
tid 19517, child 0: exiting with NO ERROR
tid 19519, child 2: exiting with NO ERROR
tid 19523, child 6: exiting with NO ERROR
tid 0, child -1: child 0 exited
tid 0, child -1: child 2 exited
tid 0, child -1: child 4 exited
tid 0, child -1: child 7 exited
tid 0, child -1: child 3 exited
tid 0, child -1: child 6 exited
tid 0, child -1: child 5 exited
tid 0, child -1: child 1 exited
:: [ 23:22:23 ] :: [   PASS   ] :: Command './walk_tree 8 /cvmfs/oasis.opensciencegrid.org' (Expected 0, got 0)
** Test PASS Score:
Uploading resultoutputfile.log .done
::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
::   Duration: 1740s
::   Assertions: 1 good, 0 bad
::   RESULT: PASS (Test)
```

Comment 6 Boyang Xue 2022-11-28 03:25:26 UTC
TEST PASS.

Reproduced on kernel-5.14.0-197.el9 (https://beaker.engineering.redhat.com/jobs/7267697)
```
[ 1104.052836] ------------[ cut here ]------------ 
[ 1104.053454] WARNING: CPU: 3 PID: 30871 at fs/fuse/readdir.c:401 fuse_parse_cache+0x13c/0x1a0 [fuse] 
[ 1104.054367] Modules linked in: tls rfkill sunrpc vfat fat drm fuse xfs libcrc32c crct10dif_ce ghash_ce sha2_ce sha256_arm64 sha1_ce virtio_net virtio_console net_failover failover virtio_blk virtio_mmio dm_mirror dm_region_hash dm_log dm_mod 
[ 1104.056985] CPU: 3 PID: 30871 Comm: walk_tree Kdump: loaded Tainted: G        W        --------- ---  5.14.0-197.el9.aarch64 #1 
[ 1104.058061] Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015 
[ 1104.058881] pstate: 20400005 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) 
[ 1104.059787] pc : fuse_parse_cache+0x13c/0x1a0 [fuse] 
[ 1104.060238] lr : fuse_parse_cache+0x8c/0x1a0 [fuse] 
[ 1104.060867] sp : ffff80000c31bc80 
[ 1104.061237] x29: ffff80000c31bc80 x28: ffff80000c31bdd0 x27: ffff00032ff60800 
[ 1104.062154] x26: ffff00048fb77058 x25: 0000000000000000 x24: ffff00048fb77000 
[ 1104.062954] x23: 0000000000000060 x22: 000000000000001c x21: ffff00048fb77040 
[ 1104.063739] x20: 0000000000000040 x19: 0000000000000038 x18: 0000000000000000 
[ 1104.064524] x17: 0000000000000000 x16: ffffaa4af116a124 x15: 0000000000000000 
[ 1104.065310] x14: 0000000000000000 x13: 0000000000000000 x12: 0000000000000000 
[ 1104.066098] x11: 0000ffffffffffff x10: 00000000000010d0 x9 : ffffaa4a7f58e19c 
[ 1104.066881] x8 : ffff0000c6a73ab0 x7 : 000000000080c121 x6 : 000000000000002e 
[ 1104.067753] x5 : 0000000000000000 x4 : 0000000000000020 x3 : 0000000000000020 
[ 1104.068731] x2 : 000000000000001c x1 : 000000000000002f x0 : ffff00048fb77058 
[ 1104.069519] Call trace: 
[ 1104.069692]  fuse_parse_cache+0x13c/0x1a0 [fuse] 
[ 1104.070316]  fuse_readdir_cached+0x26c/0x310 [fuse] 
[ 1104.070948]  fuse_readdir+0x7c/0x130 [fuse] 
[ 1104.071244]  iterate_dir+0x164/0x1b0 
[ 1104.071796]  __arm64_sys_getdents64+0x6c/0x15c 
[ 1104.072404]  invoke_syscall.constprop.0+0x7c/0xd0 
[ 1104.072731]  el0_svc_common.constprop.0+0x154/0x160 
[ 1104.073362]  do_el0_svc+0x2c/0xb0 
[ 1104.073887]  el0_svc+0x38/0x190 
[ 1104.074108]  el0t_64_sync_handler+0x9c/0x120 
[ 1104.074697]  el0t_64_sync+0x174/0x178 
[ 1104.075240] ---[ end trace fc94e5399a1dac18 ]--- 
```

Verified on kernel-5.14.0-201.el9 (https://beaker.engineering.redhat.com/jobs/7281837)
```
::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
::   Test
::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

:: [ 05:31:18 ] :: [  BEGIN   ] :: Running './walk_tree 8 /cvmfs/oasis.opensciencegrid.org'
tid 19462, child 0: alive
tid 19463, child 1: alive
tid 19464, child 2: alive
tid 19465, child 3: alive
tid 19466, child 4: alive
tid 19468, child 6: alive
tid 19470, child 7: alive
tid 19467, child 5: alive
tid 19463, child 1: exiting with NO ERROR
tid 0, child -1: child 1 exited
tid 0, child -1: child 6 exited
tid 0, child -1: child 2 exited
tid 0, child -1: child 7 exited
tid 0, child -1: child 3 exited
tid 0, child -1: child 4 exited
tid 0, child -1: child 5 exited
tid 0, child -1: child 0 exited
:: [ 06:19:44 ] :: [   PASS   ] :: Command './walk_tree 8 /cvmfs/oasis.opensciencegrid.org' (Expected 0, got 0)
** Test PASS Score:
Uploading resultoutputfile.log .done
::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
::   Duration: 2907s
::   Assertions: 1 good, 0 bad
::   RESULT: PASS (Test)
```

Comment 7 Boyang Xue 2022-11-28 03:29:05 UTC
Sorry. I just noticed it's not ON_QA yet, despite that the fix has already been merged in -201.el9. I'm reverting it back, waiting for its ON_QA.

Comment 11 errata-xmlrpc 2023-05-09 08:07:27 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: kernel security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2023:2458