Description of problem: cmdline: BOOT_IMAGE=/vmlinuz-4.5.4-300.fc24.x86_64 root=/dev/mapper/fedora-root ro rd.lvm.lv=fedora/swap rd.lvm.lv=fedora/root debug console=ttyS0,115200 console=tty0 crashkernel=128M LANG=en_US.UTF-8 abrt_version: 2.8.0 hostname: zdtm.openvz.org kernel: 4.5.4-300.fc24.x86_64 last_occurrence: 1467417743 pkg_arch: x86_64 pkg_epoch: 0 pkg_name: kernel-core pkg_release: 300.fc24 pkg_version: 4.5.4 runlevel: N 3 backtrace: :BUG: unable to handle kernel NULL pointer dereference at 0000000000000060 :IP: [<ffffffff81264f5d>] copy_tree+0x14d/0x320 :PGD 558e0067 PUD 494bb067 PMD 0 :Oops: 0000 [#1] SMP :Modules linked in: tun nf_conntrack_netlink udp_diag tcp_diag inet_diag netlink_diag af_packet_diag unix_diag binfmt_misc ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_broute bridge stp llc ebtable_filter ebtable_nat ebtables ip6table_raw ip6table_security ip6table_mangle ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_filter ip6_tables iptable_raw iptable_security iptable_mangle iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack crct10dif_pclmul crc32_pclmul ghash_clmulni_intel ppdev sbs sbshc parport_pc parport virtio_balloon acpi_cpufreq shpchp lpc_ich tpm_tis tpm btrfs xor raid6_pq loop nfsd auth_rpcgss nfs_acl lockd grace sunrpc xfs libcrc32c crc32c_intel ata_generic serio_raw pata_acpi e1000 virtio_pci : virtio virtio_ring fjes :CPU: 0 PID: 28656 Comm: criu Not tainted 4.5.4-300.fc24.x86_64 #1 :Hardware name: Parallels Software International Inc. Parallels Virtual Platform/Parallels Virtual Platform, BIOS 6.10.24198.1226784 12/09/2015 :task: ffff880023848000 ti: ffff880026fdc000 task.ti: ffff880026fdc000 :RIP: 0010:[<ffffffff81264f5d>] [<ffffffff81264f5d>] copy_tree+0x14d/0x320 :RSP: 0018:ffff880026fdfcf8 EFLAGS: 00010246 :RAX: 0000000000000000 RBX: ffff880049595200 RCX: ffff880023c2f480 :RDX: 0000000000000004 RSI: 0000000000000000 RDI: ffff880049595260 :RBP: ffff880026fdfd50 R08: ffff8800495947d0 R09: 0000000000000000 :R10: ffffe8ffffddc5d8 R11: 000000000001c5d1 R12: ffff880023c2df80 :R13: ffff88006f3a74ff R14: ffff880049595e00 R15: ffff880049595200 :FS: 00007fca9e5c7800(0000) GS:ffff88007ce00000(0000) knlGS:0000000000000000 :CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 :CR2: 0000000000000060 CR3: 00000000210e2000 CR4: 00000000000406f0 :DR0: 00007f2f1db87030 DR1: 0000000000000000 DR2: 0000000000000000 :DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000600 :Stack: : ffff880023c2e450 ffff880023c2e400 ffff88007c408000 ffff880049595e88 : 00000000de7f9542 0000000400000005 ffff880023c2e400 ffff880026f4fb40 : ffff8800239b3f00 ffffffff81c44da0 ffff8800495bf000 ffff880026fdfd90 :Call Trace: : [<ffffffff81266406>] copy_mnt_ns+0x86/0x280 : [<ffffffff810c5476>] ? create_new_namespaces+0x36/0x1b0 : [<ffffffff810c54a1>] create_new_namespaces+0x61/0x1b0 : [<ffffffff810c565d>] copy_namespaces+0x6d/0xa0 : [<ffffffff810a40f5>] copy_process.part.27+0x805/0x1810 : [<ffffffff810a52bf>] _do_fork+0xcf/0x390 : [<ffffffff810a5629>] SyS_clone+0x19/0x20 : [<ffffffff817d086e>] entry_SYSCALL_64_fastpath+0x12/0x6d :Code: 60 48 39 45 a8 4c 8d 60 a0 75 82 4c 89 f3 48 83 c4 30 48 89 d8 5b 41 5c 41 5d 41 5e 41 5f 5d c3 8b 45 cc 49 8b 75 20 85 c0 75 0e <48> 81 7e 60 80 8f 82 81 0f 84 60 01 00 00 49 8b 45 10 48 39 c1 :RIP [<ffffffff81264f5d>] copy_tree+0x14d/0x320 : RSP <ffff880026fdfcf8> :CR2: 0000000000000060 Here is a place where it crashed: static bool is_mnt_ns_file(struct dentry *dentry) { /* Is this a proxy for a mount namespace? */ return dentry->d_op == &ns_dentry_operations && ffffffff81264f5d: 48 81 7e 60 80 8f 82 cmpq $0xffffffff81828f80,0x60(%rsi) ffffffff81264f64: 81 ffffffff81264f65: 0f 84 60 01 00 00 je ffffffff812650cb <copy_tree+0x2bb> if (!(flag & CL_COPY_MNT_NS_FILE) && is_mnt_ns_file(s->mnt.mnt_root)) { s = skip_mnt_tree(s); continue;
The problem was triggered in our criu tests, I so I discovered the real problem of it all. In Linux v4.3 commit df2cf4a78e48 ("IGMP: Inhibit reports for local multicast groups") sysctl igmp_link_local_mcast_reports was introduced in ipv4_net_table. And in ipv4_net_table it's data was initialized to point on sysctl_igmp_llm_reports variable. That was so before commit 87a8a2ae65b7 ("igmp: Namespaceify igmp_llm_reports sysctl knob"). So next it's data pointer is shifted to the offset of current netnamespace relative to init_net in ipv4_sysctl_init_net function. But that is completely wrong if variable is not net-namespaced, so we get random kernel address and can write/read to/from it one int, that can lead to memory corruption and crashes in random places in kernel. So conclusion is: we can not touch /proc/sys/net/ipv4/igmp_link_local_mcast_reports in v4.3-v4.5 between those two patches. Simple reproduction(kernel-4.5.7-202.fc23.x86_64): while :; do unshare -n sysctl -w net.ipv4.igmp_link_local_mcast_reports=268435456; done
So if 87a8a2ae65b7 is the fix, then the kernel in updates-testing for F24 shouldn't have this problem. Can someone test and confirm?
> Can someone test and confirm? Checked on my F24: [root@localhost ~]# uname -r 4.6.3-300.fc24.x86_64 I ran these night: [root@localhost ~]# time while :; do unshare -n sysctl -w net.ipv4.igmp_link_local_mcast_reports=268435456; done net.ipv4.igmp_link_local_mcast_reports = 268435456 ... (for 357846 times) And everything is OK: no crash, no panic and no warnings.
Great, thank you for testing and the great analysis. Much appreciated.