Bug 1015028

Summary: Busy-looping kernel in radix_tree_next_chunk
Product: [Fedora] Fedora Reporter: Zdenek Kabelac <zkabelac>
Component: kernelAssignee: fs-maint
Status: CLOSED INSUFFICIENT_DATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: medium Docs Contact:
Priority: medium    
Version: 22CC: agk, esandeen, gansalmon, itamar, jeharris, jonathan, kernel-maint, lvm-team, madhu.chinakonda, marcelo.barbosa, rwheeler, stian, toralf.foerster
Target Milestone: ---Flags: jforbes: needinfo?
Target Release: ---   
Hardware: i686   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-11-23 17:09:45 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Zdenek Kabelac 2013-10-03 10:40:10 UTC
Description of problem:

With recent f19 kernels  lvm2 test suite is not executable since process remains stuck in endless loop:


systemd-udevd   R running      0  1006    144 0x00000084
 f3cb3e74 00000086 ffffff8e c06904ac 00000060 c0d353c0 00000000 00000012
 c0d353c0 f75ed3c0 f3d4c460 00000000 f3cb3e64 f3cb3e50 00000000 f6769560
 ffffffff 00000001 f3cb3e78 c051becf 00000000 f34748d8 0000000e 00000001
Call Trace:
 [<c06904ac>] ? radix_tree_next_chunk+0x19c/0x2b0
 [<c051becf>] ? release_pages+0x17f/0x1e0
 [<c047e70b>] __cond_resched+0x1b/0x30
 [<c0997366>] _cond_resched+0x26/0x30
 [<c051cbae>] truncate_inode_pages_range+0x15e/0x450
 [<c051cebf>] truncate_inode_pages+0x1f/0x30
 [<c058e906>] kill_bdev+0x26/0x30
 [<c058fc3f>] __blkdev_put+0x4f/0x160
 [<c05904b0>] blkdev_put+0x40/0xf0
 [<c05905ff>] blkdev_close+0x1f/0x30
 [<c0561392>] __fput+0xc2/0x1f0
 [<c05614fd>] ____fput+0xd/0x10
 [<c046e9b9>] task_work_run+0x79/0xb0
 [<c0410319>] do_notify_resume+0x59/0x90
 [<c0998c51>] work_notifysig+0x30/0x37
 [<c0990000>] ? __schedule_bug+0x45/0x60

This systemd-udev process never finishes and keeps even opened LV device opened forever.

Version-Release number of selected component (if applicable):

3.11.2-201.fc19.i686.PAE


How reproducible:
Running  lvm2 test-suite: 

in /tests  subdir executing 

make check_local T=lvconvert-thin.sh VERBOSE=1

Is typically enough to trigger endlessly looping  systemd-udevd process
(perf top shows CPU spends most of the time radix_tree_next_chunk)

Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 1 Zdenek Kabelac 2013-11-15 15:38:32 UTC
Here is much simpler way to trigger bug:


dd if=/dev/zero of=/tmp/x  bs=1M count=0 seek=1000000

losetup /dev/loop0 /tmp/x

vgcreate testvg /dev/loop0

lvcreate -s -l 100%FREE -n LV --virtualsize 64T testvg 


After this command 'systemd-udevd' starts to busy-loop in kernel
and system becomes unusable.

Comment 2 Zdenek Kabelac 2013-11-15 16:01:19 UTC
It seems to be related  to   i686 

Since the first failing size of device is  16T.


lvcreate -s -L10 -V8T  is ok on my system, but
lvcreate -s -L10 -V16T starts the loop.

Comment 3 Toralf Förster 2013-11-28 21:57:59 UTC
SInce weeks I try to bisect a hang in a 32 bit user mode linux image rleated to an endless loop in radix_tree_next_chunk :

The user mode linux dev mailing list contains more info (and lklml too) - the picture is just shown here roo : at a fuzzied UML (using trinity) the trinity process hangs here :

tfoerste@n22 ~ $ date; sudo gdb /home/tfoerste/devel/linux/linux 6199 -n -batch -ex bt
Thu Nov 28 20:59:40 CET 2013
0x0829c46f in radix_tree_next_chunk (root=0x3f, iter=0x48f57cdc, flags=6) at lib/radix-tree.c:773
773                             index &= ~((RADIX_TREE_MAP_SIZE << shift) - 1);
#0  0x0829c46f in radix_tree_next_chunk (root=0x3f, iter=0x48f57cdc, flags=6) at lib/radix-tree.c:773
#1  0x080cc78e in find_get_pages (mapping=0x499d4a58, start=0, nr_pages=14, pages=0x6) at mm/filemap.c:844
#2  0x080d654a in pagevec_lookup (pvec=0x48f57d40, mapping=0x3f, start=63, nr_pages=63) at mm/swap.c:937
#3  0x080d694a in truncate_inode_pages_range (mapping=0x499d4a58, lstart=0, lend=-1) at mm/truncate.c:241
#4  0x080d6cef in truncate_inode_pages (mapping=0x3f, lstart=25769803839) at mm/truncate.c:358
#5  0x0818c152 in ext4_evict_inode (inode=0x499d49a0) at fs/ext4/inode.c:228
#6  0x0811b46f in evict (inode=0x499d49a0) at fs/inode.c:549
#7  0x0811bf5d in iput_final (inode=<optimized out>) at fs/inode.c:1419
#8  iput (inode=0x499d49a0) at fs/inode.c:1437
#9  0x08111ec6 in do_unlinkat (dfd=5, pathname=0x8067f34 <tap_open_common+52> "\211D$\b\211T$\f\211\034$\350<\216#") at fs/namei.c:3724
#10 0x08112035 in SYSC_unlinkat (flag=<optimized out>, pathname=<optimized out>, dfd=<optimized out>) at fs/namei.c:3760
#11 SyS_unlinkat (dfd=5, pathname=134643508, flag=0) at fs/namei.c:3752
#12 0x08062ab4 in handle_syscall (r=0x49e9f1cc) at arch/um/kernel/skas/syscall.c:35
#13 0x08075115 in handle_trap (local_using_sysemu=<optimized out>, regs=<optimized out>, pid=<optimized out>) at arch/um/os-Linux/skas/process.c:198
#14 userspace (regs=0x49e9f1cc) at arch/um/os-Linux/skas/process.c:431
#15 0x0805f770 in fork_handler () at arch/um/kernel/process.c:149
#16 0x00000000 in ?? ()
tfoerste@n22 ~ $ date; sudo gdb /home/tfoerste/devel/linux/linux 6199 -n -batch -ex bt
Thu Nov 28 22:53:58 CET 2013
radix_tree_next_chunk (root=0x2, iter=0x48f57cdc, flags=18) at lib/radix-tree.c:770
770                                             if (node->slots[offset])
#0  radix_tree_next_chunk (root=0x2, iter=0x48f57cdc, flags=18) at lib/radix-tree.c:770
#1  0x080cc78e in find_get_pages (mapping=0x499d4a58, start=0, nr_pages=14, pages=0x12) at mm/filemap.c:844
#2  0x080d654a in pagevec_lookup (pvec=0x48f57d40, mapping=0x2, start=2, nr_pages=2) at mm/swap.c:937
#3  0x080d694a in truncate_inode_pages_range (mapping=0x499d4a58, lstart=0, lend=-1) at mm/truncate.c:241
#4  0x080d6cef in truncate_inode_pages (mapping=0x2, lstart=77309411330) at mm/truncate.c:358
#5  0x0818c152 in ext4_evict_inode (inode=0x499d49a0) at fs/ext4/inode.c:228
#6  0x0811b46f in evict (inode=0x499d49a0) at fs/inode.c:549
#7  0x0811bf5d in iput_final (inode=<optimized out>) at fs/inode.c:1419
#8  iput (inode=0x499d49a0) at fs/inode.c:1437
#9  0x08111ec6 in do_unlinkat (dfd=5, pathname=0x8067f34 <tap_open_common+52> "\211D$\b\211T$\f\211\034$\350<\216#") at fs/namei.c:3724
#10 0x08112035 in SYSC_unlinkat (flag=<optimized out>, pathname=<optimized out>, dfd=<optimized out>) at fs/namei.c:3760
#11 SyS_unlinkat (dfd=5, pathname=134643508, flag=0) at fs/namei.c:3752
#12 0x08062ab4 in handle_syscall (r=0x49e9f1cc) at arch/um/kernel/skas/syscall.c:35
#13 0x08075115 in handle_trap (local_using_sysemu=<optimized out>, regs=<optimized out>, pid=<optimized out>) at arch/um/os-Linux/skas/process.c:198
#14 userspace (regs=0x49e9f1cc) at arch/um/os-Linux/skas/process.c:431
#15 0x0805f770 in fork_handler () at arch/um/kernel/process.c:149
#16 0x00000000 in ?? ()

Comment 4 Zdenek Kabelac 2013-11-29 13:40:40 UTC
It's probably worth to put into this BZ reference to various related emails I've found:

http://www.spinics.net/lists/linux-fsdevel/msg55633.html

http://www.redhat.com/archives/dm-devel/2009-July/msg00131.html

http://www.redhat.com/archives/dm-devel/2009-July/msg00182.html

So as long as DIRECT read is used then  32bit arch could possibly read even bigger then 16T devices - but anything cached will fail since
32bit are simply not enough to index so many cache pages.

Unsure what is the best fix here - but busy-looping kernel isn't the best one.

Comment 5 Toralf Förster 2013-11-29 14:46:54 UTC
FWIW it is broken since few kernel version s - and with trinity and a 32 bit user mode linux image  I can reproduce it within 1/2 an hour or so - si if somebody has a good fix I can test it.

Comment 6 Justin M. Forbes 2014-01-03 22:03:58 UTC
*********** MASS BUG UPDATE **************

We apologize for the inconvenience.  There is a large number of bugs to go through and several of them have gone stale.  Due to this, we are doing a mass bug update across all of the Fedora 19 kernel bugs.

Fedora 19 has now been rebased to 3.12.6-200.fc19.  Please test this kernel update (or newer) and let us know if you issue has been resolved or if it is still present with the newer kernel.

If you have moved on to Fedora 20, and are still experiencing this issue, please change the version to Fedora 20.

If you experience different issues, please open a new bug report for those.

Comment 7 Zdenek Kabelac 2014-02-09 22:35:58 UTC
Still applies to latest vanilla 3.14....

Comment 8 Eric Sandeen 2015-02-24 16:09:28 UTC
(In reply to Zdenek Kabelac from comment #7)
> Still applies to latest vanilla 3.14....

What about more recent kernels?

Comment 9 Jaroslav Reznik 2015-03-03 15:07:20 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 22 development cycle.
Changing version to '22'.

More information and reason for this action is here:
https://fedoraproject.org/wiki/Fedora_Program_Management/HouseKeeping/Fedora22

Comment 10 Justin M. Forbes 2015-10-20 19:18:40 UTC
*********** MASS BUG UPDATE **************

We apologize for the inconvenience.  There is a large number of bugs to go through and several of them have gone stale.  Due to this, we are doing a mass bug update across all of the Fedora 22 kernel bugs.

Fedora 22 has now been rebased to 4.2.3-200.fc22.  Please test this kernel update (or newer) and let us know if you issue has been resolved or if it is still present with the newer kernel.

If you have moved on to Fedora 23, and are still experiencing this issue, please change the version to Fedora 23.

If you experience different issues, please open a new bug report for those.

Comment 11 Fedora Kernel Team 2015-11-23 17:09:45 UTC
*********** MASS BUG UPDATE **************
This bug is being closed with INSUFFICIENT_DATA as there has not been a response in over 4 weeks. If you are still experiencing this issue, please reopen and attach the relevant data from the latest kernel you are running and any data that might have been requested previously.