Bug 1740690

Created attachment 1603384 [details]
dmesg-zswap-problem-5.2.0-0.rc1.git0.1.fc31.x86_64.txt

1. Please describe the problem:

Configuring kernel to boot with zswap.enabled=1 and zswap.zpool=z3fold causes kernel to BUG.

This also makes data in swap to be inaccessible and all swap using processes become stuck. Shutdown does not to work.

[   78.257803] BUG: unable to handle page fault for address: ffffcb6203000028
[   78.258762] #PF: supervisor read access in kernel mode
[   78.260306] #PF: error_code(0x0000) - not-present page
[   78.261073] PGD 0 P4D 0 
[   78.261586] Oops: 0000 [#1] SMP PTI
[   78.262328] CPU: 0 PID: 152 Comm: kswapd0 Not tainted 5.2.8-200.fc30.x86_64 #1
[   78.263387] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
[   78.266573] RIP: 0010:z3fold_zpool_map+0x50/0xf0
[   78.268259] Code: e8 48 01 ea 0f 82 b3 00 00 00 48 c7 c3 00 00 00 80 48 2b 1d 5a b9 ec 00 48 01 d3 48 c1 eb 0c 48 c1 e3 06 48 03 1d 38 b9 ec 00 <48> 8b 53 28 83 e2 01 74 05 5b 5d 41 5c c3 48 8d 7d 10 e8 69 d8 6d
[   78.271306] RSP: 0018:ffffa9108033b868 EFLAGS: 00010286
[   78.272469] RAX: 0000000000000000 RBX: ffffcb6203000000 RCX: 0000000000000000
[   78.273603] RDX: 0000000080000000 RSI: ffff953f6bac5e18 RDI: ffff953f7c8fc600
[   78.274486] RBP: 0000000000000000 R08: ffff953f7c8fc600 R09: 0000000000000001
[   78.275356] R10: ffff953f6bac5e18 R11: ffffc9b700aef020 R12: ffff953f6bac5e18
[   78.276192] R13: ffffa9108033b8a0 R14: ffff953f45910000 R15: ffff953f7c8fc600
[   78.277158] FS:  0000000000000000(0000) GS:ffff953f7e600000(0000) knlGS:0000000000000000
[   78.278172] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   78.279182] CR2: ffffcb6203000028 CR3: 000000003583a005 CR4: 0000000000160ef0
[   78.280195] Call Trace:
[   78.280660]  zswap_writeback_entry+0x50/0x3a0
[   78.282058]  z3fold_zpool_shrink+0x37d/0x480
[   78.283096]  zswap_frontswap_store+0x2dc/0x600
[   78.283852]  __frontswap_store+0xab/0x12a
[   78.284646]  swap_writepage+0x39/0x70
[   78.285103]  pageout.isra.0+0x13c/0x350
[   78.285698]  shrink_page_list+0xc14/0xdf0
[   78.286179]  shrink_inactive_list+0x1e5/0x3c0
[   78.286693]  shrink_node_memcg+0x202/0x760
[   78.287403]  shrink_node+0xdc/0x490
[   78.288608]  balance_pgdat+0x2d1/0x510
[   78.289089]  kswapd+0x210/0x3f0
[   78.289482]  ? finish_wait+0x80/0x80
[   78.290093]  kthread+0xfb/0x130
[   78.290526]  ? balance_pgdat+0x510/0x510
[   78.291066]  ? kthread_park+0x80/0x80
[   78.291783]  ret_from_fork+0x35/0x40
[   78.292480] Modules linked in: ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ip6table_nat ip6table_mangle ip6table_raw ip6table_security iptable_nat nf_nat iptable_mangle iptable_raw iptable_security nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables vfat fat kvm_intel kvm irqbypass snd_hda_codec_generic crct10dif_pclmul ledtrig_audio crc32_pclmul ghash_clmulni_intel snd_hda_intel snd_hda_codec iTCO_wdt snd_hda_core iTCO_vendor_support snd_pcsp snd_hwdep snd_pcm snd_timer i2c_i801 snd lpc_ich soundcore virtio_net virtio_balloon joydev net_failover failover qxl serio_raw drm_kms_helper crc32c_intel ttm drm virtio_blk virtio_console qemu_fw_cfg
[   78.305499] CR2: ffffcb6203000028
[   78.305908] ---[ end trace 460d9373d78c8439 ]---
[   78.306536] RIP: 0010:z3fold_zpool_map+0x50/0xf0
[   78.307131] Code: e8 48 01 ea 0f 82 b3 00 00 00 48 c7 c3 00 00 00 80 48 2b 1d 5a b9 ec 00 48 01 d3 48 c1 eb 0c 48 c1 e3 06 48 03 1d 38 b9 ec 00 <48> 8b 53 28 83 e2 01 74 05 5b 5d 41 5c c3 48 8d 7d 10 e8 69 d8 6d
[   78.309585] RSP: 0018:ffffa9108033b868 EFLAGS: 00010286
[   78.310208] RAX: 0000000000000000 RBX: ffffcb6203000000 RCX: 0000000000000000
[   78.311049] RDX: 0000000080000000 RSI: ffff953f6bac5e18 RDI: ffff953f7c8fc600
[   78.311900] RBP: 0000000000000000 R08: ffff953f7c8fc600 R09: 0000000000000001
[   78.312777] R10: ffff953f6bac5e18 R11: ffffc9b700aef020 R12: ffff953f6bac5e18
[   78.313621] R13: ffffa9108033b8a0 R14: ffff953f45910000 R15: ffff953f7c8fc600
[   78.314420] FS:  0000000000000000(0000) GS:ffff953f7e600000(0000) knlGS:0000000000000000
[   78.315386] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   78.316069] CR2: ffffcb6203000028 CR3: 000000003583a005 CR4: 0000000000160ef0
[   78.316923] ------------[ cut here ]------------
[   78.317500] WARNING: CPU: 0 PID: 152 at kernel/exit.c:783 do_exit.cold+0x79/0x91
[   78.318391] Modules linked in: ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ip6table_nat ip6table_mangle ip6table_raw ip6table_security iptable_nat nf_nat iptable_mangle iptable_raw iptable_security nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables vfat fat kvm_intel kvm irqbypass snd_hda_codec_generic crct10dif_pclmul ledtrig_audio crc32_pclmul ghash_clmulni_intel snd_hda_intel snd_hda_codec iTCO_wdt snd_hda_core iTCO_vendor_support snd_pcsp snd_hwdep snd_pcm snd_timer i2c_i801 snd lpc_ich soundcore virtio_net virtio_balloon joydev net_failover failover qxl serio_raw drm_kms_helper crc32c_intel ttm drm virtio_blk virtio_console qemu_fw_cfg
[   78.326746] CPU: 0 PID: 152 Comm: kswapd0 Tainted: G      D           5.2.8-200.fc30.x86_64 #1
[   78.327784] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
[   78.328733] RIP: 0010:do_exit.cold+0x79/0x91
[   78.329255] Code: 42 10 48 89 44 24 18 48 8b 44 24 18 e8 51 f6 8d 00 e9 17 f0 ff ff 8b b3 c8 04 00 00 eb ad 48 c7 c7 d0 98 0c a5 e8 06 1b 06 00 <0f> 0b e9 a1 ef ff ff 48 c7 c7 e0 2d 0e a5 e8 3d a9 ff ff 90 90 90
[   78.331493] RSP: 0018:ffffa9108033bee0 EFLAGS: 00010046
[   78.332132] RAX: 0000000000000024 RBX: ffff953f7d26ddc0 RCX: 0000000000000006
[   78.333015] RDX: 0000000000000000 RSI: 0000000000000086 RDI: ffff953f7e617900
[   78.333888] RBP: 0000000000000009 R08: ffffa9108033bd95 R09: 0000000000000328
[   78.334755] R10: ffffffffa5be9b80 R11: ffffa9108033bd95 R12: 0000000000000009
[   78.335616] R13: 0000000000000009 R14: ffff953f7d26ddc0 R15: 0000000000000046
[   78.336475] FS:  0000000000000000(0000) GS:ffff953f7e600000(0000) knlGS:0000000000000000
[   78.337468] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   78.338162] CR2: ffffcb6203000028 CR3: 000000003583a005 CR4: 0000000000160ef0
[   78.339017] Call Trace:
[   78.339326]  ? kthread+0xfb/0x130
[   78.339733]  rewind_stack_do_exit+0x17/0x20
[   78.340249] ---[ end trace 460d9373d78c843a ]---

2. What is the Version-Release number of the kernel:

5.2.8-200.fc30.x86_64

3. Did it work previously in Fedora? If so, what kernel version did the issue
   *first* appear?  Old kernels are available for download at
   https://koji.fedoraproject.org/koji/packageinfo?packageID=8 :

Yes.

5.1.20-300.fc30.x86_64 seems OK
5.2.0-0.rc0.git6.1.fc31.x86_64 seems OK
5.2.0-0.rc1.git0.1.fc31.x86_64 NOT ok
5.2.1-200.fc30.x86_64 NOT ok
5.2.8-200.fc30.x86_64 NOT ok

Also note, you need to install kernels

4. Can you reproduce this issue? If so, please provide the steps to reproduce
   the issue below:

Install virtual machine from: Fedora-Workstation-netinst-x86_64-30-1.2.iso

Use "defaults" and select Minimal Install.

Add kernel parameters to /etc/default/grub: zswap.enabled=1 zswap.zpool=z3fold
You can also add "console=ttyS0" to help debugging.

Power off virtual machine

Configure virtual machine to have 4 vCPU and 1GiB memory.

Start virtual machine.

Note it has 4 CPU, about 1GiB memory and 2GiB swap configured. I don't know if these conditions are relevant. But they seem to help here to make problem to come to light.

Note zswap configuration:
$ egrep -r ^ /sys/module/zswap/parameters
/sys/module/zswap/parameters/same_filled_pages_enabled:Y
/sys/module/zswap/parameters/enabled:Y
/sys/module/zswap/parameters/max_pool_percent:20
/sys/module/zswap/parameters/compressor:lzo
/sys/module/zswap/parameters/zpool:z3fold

Run:

stress --vm $(nproc) --vm-bytes $(($(awk '/MemAvail/{print $2}' /proc/meminfo)*1024*3/2/$(nproc))) &(vmstat 1|cat -n)

Wait test to fail. Usually it takes less than 5 minutes in my setup.


5. Does this problem occur with the latest Rawhide kernel? To install the
   Rawhide kernel, run ``sudo dnf install fedora-repos-rawhide`` followed by
   ``sudo dnf update --enablerepo=rawhide kernel``:


6. Are you running any modules that not shipped with directly Fedora's kernel?:

No

7. Please attach the kernel logs. You can get the complete kernel log
   for a boot with ``journalctl --no-hostname -k > dmesg.txt``. If the
   issue occurred on a previous boot, use the journalctl ``-b`` flag.

Usually journald is swapped out during stress run and journalctl does not work. And journal does not contain relevant data.