Bug 1304850

Summary: Beagle Bone Black: mmcqd/0: page allocation failure on dnf update
Product: [Fedora] Fedora Reporter: Roman Yepishev <rye>
Component: kernelAssignee: Kernel Maintainer List <kernel-maint>
Status: CLOSED INSUFFICIENT_DATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: high Docs Contact:
Priority: unspecified    
Version: 23CC: gansalmon, itamar, jonathan, kernel-maint, madhu.chinakonda, mchehab, pbrobinson, pwhalen, rye
Target Milestone: ---Flags: rye: needinfo-
Target Release: ---   
Hardware: armv7hl   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-06-20 13:45:58 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 245418    

Description Roman Yepishev 2016-02-04 19:05:15 UTC
Description of problem:

After installation of Fedora-Server-armhfp-23-10-sda.raw.xz to the external MMC and boot from uboot provided by the image, running dnf update breaks the filesystem.


Version-Release number of selected component (if applicable):

Kernel 4.2.3-300.fc23.armv7hl (default in the image)


How reproducible:

Always


Steps to Reproduce:

1. Write Fedora-Server-armhfp-23-10-sda.raw.xz to an SD card
2. Boot BeagleBoneBlack while pressing S2 switch (to use uboot from SD card)
3. Perform initial configuration with display attached - languate, root password, etc.
4. Run dnf update


Actual results:

[ 1977.215963] mmcqd/0: page allocation failure: order:4, mode:0xc020
[ 1977.216205] CPU: 0 PID: 552 Comm: mmcqd/0 Not tainted 4.2.3-300.fc23.armv7hl #1
[ 1977.216397] Hardware name: Generic AM33XX (Flattened Device Tree)
[ 1977.216620] [<c02184b0>] (unwind_backtrace) from [<c0213584>] (show_stack+0x18/0x1c)
[ 1977.216848] [<c0213584>] (show_stack) from [<c08d71d0>] (dump_stack+0x74/0x90)
[ 1977.217059] [<c08d71d0>] (dump_stack) from [<c03307a4>] (warn_alloc_failed+0xf0/0x11c)
[ 1977.217293] [<c03307a4>] (warn_alloc_failed) from [<c0333bb4>] (__alloc_pages_nodemask+0x7a8/0x850)
[ 1977.217533] [<c0333bb4>] (__alloc_pages_nodemask) from [<c0333ea0>] (alloc_kmem_pages+0x68/0xfc)
[ 1977.217777] [<c0333ea0>] (alloc_kmem_pages) from [<c034c02c>] (kmalloc_order+0x18/0x28)
[ 1977.217999] [<c034c02c>] (kmalloc_order) from [<c034c05c>] (kmalloc_order_trace+0x20/0x80)
[ 1977.218233] [<c034c05c>] (kmalloc_order_trace) from [<c05b6c30>] (edma_prep_slave_sg+0xc8/0x284)
[ 1977.218698] [<c05b6c30>] (edma_prep_slave_sg) from [<bf0da948>] (omap_hsmmc_request+0x368/0x470 [omap_hsmmc])
[ 1977.219158] [<bf0da948>] (omap_hsmmc_request [omap_hsmmc]) from [<bf074fe0>] (mmc_start_request+0x234/0x274 [mmc_core])
[ 1977.219591] [<bf074fe0>] (mmc_start_request [mmc_core]) from [<bf07609c>] (mmc_start_req+0x2a8/0x32c [mmc_core])
[ 1977.219961] [<bf07609c>] (mmc_start_req [mmc_core]) from [<bf13e2e0>] (mmc_blk_issue_rw_rq+0x414/0xa04 [mmc_block])
[ 1977.220271] [<bf13e2e0>] (mmc_blk_issue_rw_rq [mmc_block]) from [<bf13ecb8>] (mmc_blk_issue_rq+0x3e8/0x454 [mmc_block])
[ 1977.220572] [<bf13ecb8>] (mmc_blk_issue_rq [mmc_block]) from [<bf13f460>] (mmc_queue_thread+0xd4/0x168 [mmc_block])
[ 1977.220857] [<bf13f460>] (mmc_queue_thread [mmc_block]) from [<c02669c4>] (kthread+0xe0/0xf4)
[ 1977.221106] [<c02669c4>] (kthread) from [<c020fa08>] (ret_from_fork+0x14/0x2c)
[ 1977.221303] Mem-Info:
[ 1977.221400] active_anon:11857 inactive_anon:20158 isolated_anon:0
                active_file:33763 inactive_file:34101 isolated_file:0
                unevictable:0 dirty:2050 writeback:6891 unstable:0
                slab_reclaimable:7612 slab_unreclaimable:8511
                mapped:5559 shmem:40 pagetables:426 bounce:0
                free:2845 free_pcp:59 free_cma:1714
[ 1977.222246] DMA free:11380kB min:2840kB low:3548kB high:4260kB active_anon:47428kB inactive_anon:80632kB active_file:135052kB inactive_file:136404kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:523264kB managed:505504kB mlocked:0kB dirty:8200kB writeback:27564kB mapped:22236kB shmem:160kB slab_reclaimable:30448kB slab_unreclaimable:34044kB kernel_stack:872kB pagetables:1704kB unstable:0kB bounce:0kB free_pcp:236kB local_pcp:236kB free_cma:6856kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[ 1977.223302] lowmem_reserve[]: 0 0 0 0
[ 1977.223563] DMA: 589*4kB (UMC) 470*8kB (UEMC) 185*16kB (UEMC) 60*32kB (UMC) 4*64kB (C) 1*128kB (C) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB = 11380kB
[ 1977.224197] 68043 total pagecache pages
[ 1977.224311] 139 pages in swap cache
[ 1977.224413] Swap cache stats: add 1364, delete 1225, find 105/161
[ 1977.224568] Free swap  = 495356kB
[ 1977.224660] Total swap = 500236kB
[ 1977.224753] 130816 pages RAM
[ 1977.224838] 0 pages HighMem/MovableOnly
[ 1977.224940] 344 pages reserved
[ 1977.225026] 4096 pages cma reserved
[ 1977.225143] edma-dma-engine edma-dma-engine.0: edma_prep_slave_sg: Failed to allocate a descriptor
[ 1977.225377] omap_hsmmc 48060000.mmc: prep_slave_sg() failed
[ 1977.225528] omap_hsmmc 48060000.mmc: MMC start dma failure
[ 1977.227260] mmcblk0: unknown error -1 sending read/write command, card status 0x900
[ 1977.277269] mmc0: tried to reset card
[ 1977.284137] mmcqd/0: page allocation failure: order:4, mode:0xc020

Eventually the system becomes so unstable that even dmesg fails with Segmentation Fault.



Expected results:

Update completes successfully.



Additional info:

This seems to be related - https://bugs.launchpad.net/ubuntu/+source/linux-ti-omap4/+bug/746137

Setting vm.min_free_kbytes = 8192 via sysctl makes the error go away and update completes ~successfully (if kernel package is not excluded, then the system will not boot, but that seems to be a different issue I am still investigating).

Comment 1 Peter Robinson 2016-02-05 09:53:52 UTC
what sort of SD card do you have, we've found that some really don't cope well with the running of a standard filesystem/distro and I suspect it might well be a HW issue

Comment 2 Roman Yepishev 2016-02-05 14:55:11 UTC
Sure, I had the same idea. Originally failure occured on a brand new SD card (Samsung microSDHC UHS-I - class 10 card).

Then I reverted to the older Kingston 16GB SDC10 card (also class 10) - it went through a bunch of heavy I/O operations before with Ubuntu/Debian (updates, syncthing, dd write tests), but I've started getting page allocation failures once I let dnf do it's thing on Fedora 23. Both cards started to work fine once I adjusted vm.min_free_kbytes.

Comment 3 Josh Boyer 2016-05-27 13:33:20 UTC
Is this still an issue with 4.5.y?

Comment 4 Roman Yepishev 2018-02-19 17:53:36 UTC
Leaving this closed and clearning needinfo flag as I no longer have the device in question to check newer kernels.