Bug 1401012

[a little more detail and links to the live threads]

I am fighting the same issue after upgrading a KVM guest to F25. It appears you have the same problem (many) others are having with still using a PAE kernel.

Seems to be recent work on a fix
https://bugzilla.redhat.com/show_bug.cgi?id=1373339

An older ticket from when the problem seems to have started
https://bugzilla.redhat.com/show_bug.cgi?id=1075185

For my testing I have munin logging the guest and I am now at only 3GB of 10GB of (guest assigned) memory, no swap used... and rsync, Tomcat, and anything else that moves now gets nailed. 
I am testing:
echo 1 > /proc/sys/vm/overcommit_memory

All my 64bit hosts and guests are fine. Only the busy 32bit PAE guest is getting OOM killed.

This is still happening on kernel-PAE-4.9.0-1.fc26.i686

32-bit x86 is a low priority for the Fedora kernel team and relies on greater community effort for support.  You are more likely to get feedback by reporting issues directly upstream.

I also see this when running rsync to a f24 32 bit pae or non-pae target.

a folder with a large tree always dies on the destination with the OOM.

early f24 32 kernels did not do it.

This may be the bug I am seeing (large dir scans cause oom-killer then hang/panic on 32-bit PAE).  I have git bisected my bug in the kernel to commit b2e18757f2c9d1cdd746a882e9878852fdec9501
Author: Mel Gorman <mgorman>
Date:   Thu Jul 28 15:45:37 2016 -0700
mm, vmscan: begin reclaiming pages on a per-node basis

I have posted about it to the LKML and those guys are helping out, some work has already been done on it and I am testing their patches now.  Keep an eye on the LKML thread I started:
https://lkml.org/lkml/2017/1/11/182

I'll report back if/when a patch fixes the problem and hopefully someone here in Fedoraland can apply the patch to a Fedora 24 errata kernel.

Trevor: from your LKML post:
>  I tuned the RAM down because around 8GB the PAE kernel has massive IO speed issues

I suggest trying "echo 1 >/proc/sys/vm/highmem_is_dirtyable" as per
https://bugzilla.redhat.com/show_bug.cgi?id=1373339#c24 which made a huge difference for me.

Trevor: also:
> [it's] rsync or rdiff-backup also doing big dir scans

In my case it was rsync.

> when I do "find /" manually I can't trigger the bug.

I think it's the file reads, not directory scans.  If that's the case, execing dd from find to read the files should trigger it:

find / -type f -exec dd of=/dev/null if='{}' \;

The kernel guys (Mel & Michal) have been working with me a lot on this issue, but so far no patch solves the problem.  We are continuing, and those interested can google the LKML thread referenced in comment #5.

However, we seem to have a workaround that everyone here can employ immiedately:

add to your kernel boot options:
cgroup_disable=memory

Usually you'd put this in the /etc/default/grub file in the GRUB_CMDLINE_LINUX section, and auto-regen your grub.cfg and reboot.

With that option my testing didn't crash over 3 nights, when 90% of the time it crashes in 1 night and 100% within 2 nights.

I'm hazy as to what the option actually does but I suppose it's relatively harmless, at least compared to the oom/panic alternative.

I'll post again when we have a real patch to solve this bug.

on a 4.8.16-200 PAE the cgroup_disable=memory didn't help - still get OOM kills when rsync tries to scan a very large folder (ext3, dir_index) for updating.

Hmm, did you confirm the cgroup_disable=memory actually took effect by checking /proc/cmdline?

Perhaps it also has to do with the amount of RAM.  If yours is very large, try constraining it to something like 6G with another boot option: mem=6G.  PAE gets worse the more RAM you have.  On my box with 16GB if I constrain it to 6G and do the cgroup_disable=memory then it didn't oom or crash for 3+ days.  Without cgroup_disable=memory it oom/crashes in 1 day 90% of the time, and always within 2 days.

I'll report back when I get some more solutions/ideas.  However, it definitely appears that if you want >4GB RAM you need to think about switching to a 64-bit OS.  I'm still trying to figure out a way to trick dnf into allowing an "upgrade" to 64-bit (kernel and userland).  Complete wipe/reinstalls would be a major pain.

Certainly, the cgroup_disable=memory doesn't stop all OOM killing, and maybe it shouldn't.  Remember that the 4.7 kernel totally reworked the OOM management (for good or bad), and while this issue is one that affects only 32bit systems, there are lots of other valid reasons for OOM to occur.  I know that I'm also seeing 1 or 2 a week on 64bit systems, where previously I didn't see any.

With regard to "upgrading" to a 64-bit system, I've done it in the past (somewhere around F10 with yum) and you can do it, with lots of effort.  I know I needed to put together a few scripts to edit RPM lists.  Even now I still find a few funny RPMs still lurking around.

We have apparently the same issue on many boxes with kernel-PAE after upgrading
to f25 (possibly the issue is related to kernels 4.8 or 4.9).
The problem is systematic on a computer that we use to do large backups (with rsync), however it also occasionally happens with large system upgrades using
dnf.

No problem ad all with kernels up to 4.7.

I tried adding

cgroup_disable=memory

to the kernel boot options, but it did not help.  This is a snippet from
/var/log/messages:

[...]
Jan 28 21:29:32 solaris kernel: Linux version 4.9.5-200.fc25.i686+PAE (mockbuild.fedoraproject.org) (gcc version 6.3.1 20161221 (Red Hat 6.3.1-1) (GCC) ) #1 SMP Fri Jan 20 12:43:13 UTC 2017
[...]
Jan 28 21:29:32 solaris kernel: Kernel command line: BOOT_IMAGE=/boot/vmlinuz-4.9.5-200.fc25.i686+PAE root=UUID=a7916f45-3648-405b-b127-9ab6044d566b ro LANG=en_US.UTF-8 cgroup_disable=memory
[...]
Jan 28 21:33:40 solaris kernel: rsync invoked oom-killer: gfp_mask=0x25000c0(GFP_KERNEL_ACCOUNT), nodemask=0, order=0, oom_score_adj=0
Jan 28 21:33:40 solaris kernel: rsync cpuset=/ mems_allowed=0
Jan 28 21:33:40 solaris kernel: CPU: 0 PID: 1432 Comm: rsync Tainted: G        W       4.9.5-200.fc25.i686+PAE #1
Jan 28 21:33:40 solaris kernel: Hardware name: Dell Inc. PowerEdge R320/0KM5PX, BIOS 2.4.2 01/29/2015
Jan 28 21:33:40 solaris kernel: e8ddbc48 dcb6c6c0 e8ddbd74 f6b43600 e8ddbc78 dc9e7c10 00000206 e8ddbc78
Jan 28 21:33:40 solaris kernel: dcb7237f dd273940 e8ddbc7c eabdb180 dcf889b4 f6b43600 dd17cb55 e8ddbd74
Jan 28 21:33:40 solaris kernel: e8ddbcbc dc9810bd dc87717a e8ddbca8 dc980d49 00000003 00000000 000000b7
Jan 28 21:33:40 solaris kernel: Call Trace:
Jan 28 21:33:40 solaris kernel: [<dcb6c6c0>] dump_stack+0x58/0x78
Jan 28 21:33:40 solaris kernel: [<dc9e7c10>] dump_header+0x64/0x1a6
Jan 28 21:33:40 solaris kernel: [<dcb7237f>] ? ___ratelimit+0x9f/0x100
Jan 28 21:33:40 solaris kernel: [<dc9810bd>] oom_kill_process+0x1fd/0x3c0
Jan 28 21:33:40 solaris kernel: [<dc87717a>] ? has_capability_noaudit+0x1a/0x30
Jan 28 21:33:40 solaris kernel: [<dc980d49>] ? oom_badness.part.13+0xc9/0x140
Jan 28 21:33:40 solaris kernel: [<dc981585>] out_of_memory+0xf5/0x2b0
[...]

Maurizio, how much RAM do you have?  It seems for cgroup_disable=memory to work you also need to limit your RAM.  Also add to the boot command mem=6G (or really anything 2-6G, I have good success with 3G, and also 6G).  Confirm with top or meminfo once you're booted.  The guys who cgroup_disable=memory doesn't work for I'm betting have gobs of RAM, certainly >8G.  All PAE problems (oom and I/O problems especially) get worse the more RAM you have.  If someone here has it still oom/crash with the option on AND RAM limited, I'd be shocked.

Linus (and others) has already said you shouldn't run PAE with >4G.  I run a ton of systems with PAE >4G just because going onsite to put heads on them to wipe/reinstall them to 64-bit will be a major undertaking.  However, once I figure out a way to headless upgrade (if possible), I'm going to do it.  It's obvious that PAE is the ugly stepchild that Linus hates, RH says they don't support (much) (see comment #3), and gets almost no testing by kernel devs compared to 64-bit.  This is the 4th kernel bug that is PAE-only that I've bisected/lkml'd in the last year and it's getting to be a pain.  I'm just a lowly user, not a LKML wizard.

I'm sure the main reason why everyone here is using PAE is like me: because we've just upgraded these boxes (which long ago were 32-bit only) while never reinstalling from scratch.  I suppose some might have 32-bit only apps/drivers but even that's not an imperative reason anymore.  I kind of wish Linus would just come out and say "PAE shouldn't be run and is unsupported on RAM >xGB" where X is whatever (4 or 8GB).  "Pretending" it's a normal, supported, tested, OS is now just a myth these days.  Yet that's the impression we all get: "Oh you have 64GB of RAM, PAE is perfect!" when it's not.

The LKML thread is ongoing (~30 emails already), and I'm doing a ton of testing.  A fix patch may actually be out there already, I just need to confirm it.
https://lkml.org/lkml/2017/1/11/182

Right.  I have 16Gb of RAM.  I just rebooted with the addition of mem=6G:
# cat /proc/cmdline 
BOOT_IMAGE=/boot/vmlinuz-4.9.5-200.fc25.i686+PAE root=UUID=a7916f45-3648-405b-b127-9ab6044d566b ro LANG=en_US.UTF-8 cgroup_disable=memory mem=6G
(verified with top).
I tested the backup script... and this is the outcome:

----------------------- /var/log/messages --------------------
Jan 28 23:38:52 solaris systemd-journald: System journal (/var/log/journal/) is 816.1M, max 1.9G, 1.1G free.
Jan 28 23:38:55 solaris kernel: rsync cpuset=/ mems_allowed=0
Jan 28 23:38:55 solaris kernel: CPU: 1 PID: 1562 Comm: rsync Tainted: G        W       4.9.5-200.fc25.i686+PAE #1
Jan 28 23:38:55 solaris kernel: Hardware name: Dell Inc. PowerEdge R320/0KM5PX, BIOS 2.4.2 01/29/2015
Jan 28 23:38:55 solaris kernel: ef8d9ba4 ca76c6c0 ef8d9cd0 f0f99200 ef8d9bd4 ca5e7c10 00000206 ef8d9bd4
Jan 28 23:38:55 solaris kernel: ca77237f cae73940 ef8d9bd8 f6b598c0 cab889b4 f0f99200 cad7cb55 ef8d9cd0
Jan 28 23:38:55 solaris kernel: ef8d9c18 ca5810bd ca47717a ef8d9c04 ca580d49 00000003 00000000 0000015b
Jan 28 23:38:55 solaris kernel: Call Trace:
Jan 28 23:38:55 solaris kernel: [<ca76c6c0>] dump_stack+0x58/0x78
Jan 28 23:38:55 solaris kernel: [<ca5e7c10>] dump_header+0x64/0x1a6
Jan 28 23:38:55 solaris kernel: [<ca77237f>] ? ___ratelimit+0x9f/0x100
Jan 28 23:38:55 solaris kernel: [<ca5810bd>] oom_kill_process+0x1fd/0x3c0
Jan 28 23:38:55 solaris kernel: [<ca47717a>] ? has_capability_noaudit+0x1a/0x30
Jan 28 23:38:55 solaris kernel: [<ca580d49>] ? oom_badness.part.13+0xc9/0x140
[...]
Jan 28 23:38:57 solaris kernel: Mem-Info:
Jan 28 23:38:57 solaris kernel: active_anon:7386 inactive_anon:165 isolated_anon:0#012 active_file:65552 inactive_file:1001287 isolated_file:64#012 unevictable:0 dirty:0 writeback:0 unstable:0#012 slab_reclaimable:189633 slab_unreclaimable:10445#012 mapped:15763 shmem:271 pagetables:593 bounce:0#012 free:61656 free_pcp:741 free_cma:0
Jan 28 23:38:57 solaris kernel: Node 0 active_anon:29544kB inactive_anon:660kB active_file:262208kB inactive_file:4005148kB unevictable:0kB isolated(anon):0kB isolated(file):256kB mapped:63052kB dirty:0kB writeback:0kB shmem:1084kB writeback_tmp:0kB unstable:0kB pages_scanned:11348367 all_unreclaimable? yes
Jan 28 23:38:57 solaris kernel: DMA free:3184kB min:68kB low:84kB high:100kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15980kB managed:15904kB mlocked:0kB slab_reclaimable:12420kB slab_unreclaimable:300kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Jan 28 23:38:57 solaris kernel: lowmem_reserve[]: 0 780 5223 5223
Jan 28 23:38:57 solaris kernel: Normal free:3484kB min:3536kB low:4420kB high:5304kB active_anon:0kB inactive_anon:0kB active_file:1028kB inactive_file:0kB unevictable:0kB writepending:0kB present:892920kB managed:815220kB mlocked:0kB slab_reclaimable:746112kB slab_unreclaimable:41480kB kernel_stack:1392kB pagetables:0kB bounce:0kB free_pcp:1920kB local_pcp:392kB free_cma:0kB
Jan 28 23:38:57 solaris kernel: lowmem_reserve[]: 0 0 35543 35543
Jan 28 23:38:57 solaris kernel: HighMem free:239956kB min:512kB low:5544kB high:10576kB active_anon:29544kB inactive_anon:660kB active_file:261044kB inactive_file:4005096kB unevictable:0kB writepending:0kB present:4549576kB managed:4549576kB mlocked:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:2372kB bounce:0kB free_pcp:1044kB local_pcp:108kB free_cma:0kB
Jan 28 23:38:57 solaris kernel: lowmem_reserve[]: 0 0 0 0
Jan 28 23:38:57 solaris kernel: DMA: 12*4kB (UE) 8*8kB (UE) 4*16kB (UE) 10*32kB (U) 6*64kB (UM) 2*128kB (U) 2*256kB (UM) 1*512kB (M) 1*1024kB (M) 0*2048kB 0*4096kB = 3184kB
Jan 28 23:38:57 solaris kernel: Normal: 93*4kB (UMEH) 45*8kB (MEH) 102*16kB (UMEH) 25*32kB (UMH) 5*64kB (U) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3484kB
Jan 28 23:38:57 solaris kernel: HighMem: 273*4kB (UM) 340*8kB (UM) 175*16kB (UM) 66*32kB (UM) 41*64kB (UM) 8*128kB (UM) 9*256kB (UM) 14*512kB (U) 7*1024kB (U) 5*2048kB (U) 49*4096kB (UM) = 239956kB
Jan 28 23:38:58 solaris kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Jan 28 23:38:58 solaris kernel: 1067131 total pagecache pages
Jan 28 23:38:58 solaris kernel: 0 pages in swap cache
Jan 28 23:38:58 solaris kernel: Swap cache stats: add 0, delete 0, find 0/0
Jan 28 23:38:58 solaris kernel: Free swap  = 8388604kB
Jan 28 23:38:58 solaris kernel: Total swap = 8388604kB
Jan 28 23:38:58 solaris kernel: 1364619 pages RAM
Jan 28 23:38:58 solaris kernel: 1137394 pages HighMem/MovableOnly
Jan 28 23:38:58 solaris kernel: 19444 pages reserved
Jan 28 23:38:58 solaris kernel: 0 pages hwpoisoned
Jan 28 23:38:58 solaris kernel: [ pid ]   uid  tgid total_vm      rss nr_ptes nr_pmds swapents oom_score_adj name
Jan 28 23:38:58 solaris kernel: [  972]     0   972    10059     3770      14       3        0             0 systemd-journal
Jan 28 23:38:58 solaris kernel: [  996]     0   996     4039      545       8       3        0             0 lvmetad
Jan 28 23:38:58 solaris kernel: [ 1009]     0  1009     3640     1176       8       3        0         -1000 systemd-udevd
Jan 28 23:38:58 solaris kernel: [ 1088]     0  1088     3492      752       6       3        0         -1000 auditd
Jan 28 23:38:58 solaris kernel: [ 1109]     0  1109     1470      950       6       3        0             0 smartd
[...]
Jan 28 23:38:59 solaris kernel: Out of memory: Kill process 1116 (rsyslogd) score 2 or sacrifice child
Jan 28 23:38:59 solaris kernel: Killed process 1116 (rsyslogd) total-vm:754420kB, anon-rss:572kB, file-rss:29460kB, shmem-rss:0kB
Jan 28 23:38:59 solaris kernel: systemd-journal invoked oom-killer: gfp_mask=0x2420848(GFP_NOFS|__GFP_NOFAIL|__GFP_HARDWALL|__GFP_MOVABLE), nodemask=0, order=0, oom_score_adj=0
[...]

It is quite frustrating... I had to reinstall an old f23 kernel (4.4.7-300.fc23.i686+PAE) and boot from it to have my backups working as expected :-(

Very weird.  There must be some other mitigating factor on my boxes that allow it to not oom with a similar cmdline.

You also seem to be able to make it oom on demand and I have yet to do that... I have to wait overnight for some regular jobs to hit before it ooms for me.  What exactly are you running (exact command/config).  It would be so much easier for me to debug this problem if I could reproduce it on demand in minutes, not days.

Also, when yours oom's, does it oom and recover, or oom a bunch of times until the system hangs?  I get actual hangs/freezes about 75% of the time.  Sometimes I can reboot before it hangs completely.  Rare times it just semi-recovers on its own.

I *may* have a good 4.9 or 4.10 kernel finally I'm testing right now that I can send you link / build instructions for if you want to git and compile yourself.  I'll know for sure after tonight if we finally have patches that solve this problem.  It would actually be very helpful if you guys who have the problem "worse" than me can git/build and confirm it solves the bug on your boxes too.

4.7.10 (F23) is the last kernel that doesn't have this bug in Fedora-land AFAIK.  That's the one I boot to when I need it stable.  It never exhibits this bug.  4.8.8 was the first one I used that did (I never tried Fedora-ized 4.8.0-4.8.7 but from my reading/bisecting I'm pretty sure the problem started at 4.8.0).  Of course 4.4.7 would be bug-free too, if that's the only one you can get your hands on.

This the bash script that (usually) causes the oom:

---------------------------------------------------------------
#!/bin/bash

hostname=`hostname`
echo "$hostname: $0"

mp=/mnt/cantor_home_backup
mount -o nfsvers=3 192.168.1.196:/exports/home ${mp}

if [ $? -eq 0 ] ; then

  rsync -aqol ${mp}/matem/ /home/backup/home-backup/matem
  rsync -aqol ${mp}/fisica/ /home/backup/home-backup/fisica
  rsync -aqol ${mp}/tesi/ /home/backup/home-backup/tesi
  rsync -aqol ${mp}/staff/ /home/backup/home-backup/staff
  rsync -aqol ${mp}/pc/ /home/backup/home-backup/pc
  rsync -aqol ${mp}/informatica/ /home/backup/home-backup/informatica
  rsync -aqol ${mp}/scienzeamb/ /home/backup/home-backup/scienzeamb
  rsync -aqol ${mp}/profiles/ /home/backup/home-backup/profiles
  umount ${mp}
else
  echo "Unable to perform home backup"
fi
----------------------------------------------------------------

So it involves an NFS mount (version 3, because version 4 has its own problems)
and an rsync.
The source folders are large home directories:

# du -sxh *
13G     fisica
405M    informatica
151G    matem
177G    pc
35M     profiles
148K    scienzeamb
8.2G    staff
4.0K    temp
65G     tesi

Caching of NFS-filesystem data in RAM might be a trigger for the problem...

The LKML guys, specifically Michal Hocko, seem to have produced a working fix!  Their latest attempt has survived 3 nights on my box without any cgroup/mem boot options, and for me that means the bug is solved.

However, since others here said that the cgroup/mem boot option didn't help them at all (when it did for me), it might be prudent if another tester could test the fix and report back if this really solves this bug, or if it just mitigates it on my box for whatever reason.

You'll need to compile the kernel from a git tree.  It's not too hard.

git clone git://git.kernel.org/pub/scm/linux/kernel/git/mhocko/mm.git MHocko
cd MHocko
git checkout since-4.9

# get a Fedora-ized config file, you can use mine (I found it really tricky to get the right config for PAE on a 64-bit capable box):
https://tecnopolis.ca/LinuxTest/dotconfig
# save it in the MHocko directory as .config
# you shouldn't have to answer more than a few config questions, or something isn't working right.

# compile everything:
nice make bzImage && nice make modules

# install
nice make modules_install && nice make install

If you get errors during the compile then you need to dnf install some more dev tools / libraries.  Try:
dnf builddep --downloadonly kernel
See what it wants to do and then take out the --downloadonly to actually install.  I think sometimes you need a couple more things, the errors will guide you.  Warning, the build can take 2+ hours!

Change your grub2.cfg to boot from the newly installed kernel and reboot.  Remove the cgroup/memory boot options.  Hit it with your oom-causing things as much as you can.  Report back here!

Since this fixes it for me, I'm trying to get the patches into 4.9 stable so F24 can pick it up.  Good luck everyone!

I tried out the MHocko fixed kernel... Everything seems fine :-) My usual "rsync"
backup completed with no problems and I am now forcing an rsync onto an empty
directory just to stress the system more.
"top" seems normal.  At the moment I see:

KiB Mem : 16542660 total,  7949088 free,   108540 used,  8485032 buff/cache
KiB Swap:  8388604 total,  8388604 free,        0 used. 15797028 avail Mem

Let's hope now that this patch gets into the next kernel-PAE update!

Thank you Travor (and the kernel guys) for your *enormous* effort into the problem

I will add a new entry here as soon as the full rsync has completed.

rsync *did* complete without problems... This is a quite *strong* indication that
the proposed patch solves the problem!

The LKML guys have solved this problem and I have tested their newest/final(?) fix extensively.

Michal reports:
"I will send this patch to 4.9+ stable as soon as it hits Linus tree."

That means we should be able to get this into F24 (please please!) and F25 soon-ish, once it comes down from upstream.  If anyone wants the details or to compile one of the test kernels for early testing, google this:
"mm, vmscan: commit makes PAE kernel crash nightly" site:lkml.org

Great news!  I also tested the proposed patch on my kernel-PAE box with
F25.  It used to OOM every night during a backup rsync (involving
more than 200Gb of data, now with the 'fixed' 4.9 kernel is running
smoothly since six days ago.

I am looking forward for an updated f25 kernel-PAE... since the same problem
hits randomly also other computers in our department and I cannot fix all of
them "by hands"

F24 4.9.7-101 i686 PAE i am unable to trigger the OOM bug.

My thanks as well.

Mark, I'm not sure if 4.9.7.101 has the fix for our bug in it or not.  I'm guessing it doesn't?  So be careful as it may still blow up on you.  Maybe a Fedora kernel maintainer can comment.  It seems to be quite difficult using the web interfaces to find out what patches are in a specific errata release, unless a bz is specifically linked to the release.

Mmh, I will shortly check kernel-PAE-core-4.9.8-201.fc25.i686.rpm
on our box.  Last that we checked was kernel-PAE-core-4.9.5-200.fc25.i686
(it crashed with the OOM bug).

Nope.  kernel-PAE-core-4.9.8-201.fc25.i686 just crashed with the usual OOM
one minute ago (it took just a few minutes of an rsync to jump up with the
kwapd0 CPU usage and then I lost control.  After reboot with a remote "racadm serveraction powercycle" from the control interface I can verify that it OOMed.

For the records:

Feb 11 19:19:39   boot into kernel-PAE-core-4.9.8-201.fc25.i686
Feb 11 19:24:55   systemd-journal invoked oom-killer...

in /var/log/messages I find a strange set of lines:

Feb 11 19:19:39 solaris kernel: ------------[ cut here ]------------
Feb 11 19:19:39 solaris kernel: WARNING: CPU: 0 PID: 0 at arch/x86/kernel/apic/apic.c:2065 __generic_processor_info+0x28c/0x360
Feb 11 19:19:39 solaris kernel: Only 31 processors supported.Processor 32/0x60 and the rest are ignored.
Feb 11 19:19:39 solaris kernel: Modules linked in:
Feb 11 19:19:39 solaris kernel: CPU: 0 PID: 0 Comm: swapper Not tainted 4.9.8-201.fc25.i686+PAE #1
Feb 11 19:19:39 solaris kernel: Hardware name: Dell Inc. PowerEdge R320/0KM5PX, BIOS 2.4.2 01/29/2015
Feb 11 19:19:39 solaris kernel: ce25be30 cdb6c8c0 ce25be74 ce161070 ce25be60 cd86c95a ce1591c0 ce25be94
Feb 11 19:19:39 solaris kernel: 00000000 ce161070 00000811 cd8498fc 00000811 00000060 00000015 00000020
Feb 11 19:19:39 solaris kernel: ce25be80 cd86c9c6 00000009 00000000 ce25be74 ce1591c0 ce25be94 e86116e0
Feb 11 19:19:39 solaris kernel: Call Trace:
Feb 11 19:19:39 solaris kernel: [<cdb6c8c0>] dump_stack+0x58/0x78
[ --- 15 similar lines removed --- ]
Feb 11 19:19:39 solaris kernel: ---[ end trace ea3830f176c360f7 ]---


But this machine has only 4 processors (from /proc/cpuinfo)

I just tried the new kernel 4.9.10-200.fc25.i686+PAE
but the problem is still there: rsync produces oom:

# grep oom-killer /var/log/messages
Feb 21 14:53:02 solaris kernel: rsync invoked oom-killer: gfp_mask=0x2420848(GFP_NOFS|__GFP_NOFAIL|__GFP_HARDWALL|__GFP_MOVABLE), nodemask=0, order=0, oom_score_adj=0
Feb 21 14:54:30 solaris kernel: rsync invoked oom-killer: gfp_mask=0x24000c0(GFP_KERNEL), nodemask=0, order=0, oom_score_adj=0
Feb 21 14:55:50 solaris kernel: jbd2/sda4-8 invoked oom-killer: gfp_mask=0x2420848(GFP_NOFS|__GFP_NOFAIL|__GFP_HARDWALL|__GFP_MOVABLE), nodemask=0, order=0, oom_score_adj=0

:-(

I'm pretty sure the commit that fixes this bug is NOT in any vanilla yet.  There was a notice to LKML by a bot that the commit causes a 11% performance hit on FS performance, so maybe it's all delayed until that is resolved.  Until then, we must use the workarounds or our own custom-built kernels.

*********** MASS BUG UPDATE **************

We apologize for the inconvenience.  There is a large number of bugs to go through and several of them have gone stale.  Due to this, we are doing a mass bug update across all of the Fedora 25 kernel bugs.

Fedora 25 has now been rebased to 4.10.9-200.fc25.  Please test this kernel update (or newer) and let us know if you issue has been resolved or if it is still present with the newer kernel.

If you have moved on to Fedora 26, and are still experiencing this issue, please change the version to Fedora 26.

If you experience different issues, please open a new bug report for those.

Unfortunately I am no longer able to reproduce the problem.  A few weeks ago we moved our file server on a new hardware and reinstalled the OS.  This is the
nfs server used as a client from the box experiencing the OOM problem during an
rsync.

Since that upgrade we did never experience the OOM (let me stress it again: we
did not upgrade the box experiencing the OOM, we upgraded the box that is backupped via rsync via NFS mount on the machine that did experience the OOM).

The problem did not occur with the kernel 4.9.10-200.fc25.i686+PAE
(see Comment 29 above) nor with the current 4.10.6-200.fc25.i686+PAE
kernel.  I will soon test the newer version, but I have a strong feeling that we shall not see the problem again.

It seems it was due to some strange combination of ingredients that includes
the specific NFS version on the client box, or perhaps it was due to some
particular timing or whatever.

After upgrading from 4.9.5 to 4.10.8 I did not see any OOM anymore.

Excellent. Closing this as fixed, if others find that it is not, feel free to reopen.