Bug 1352177
| Summary: | :BUG: unable to handle kernel NULL pointer dereference in copy_tree+0x14d/0x320 | ||
|---|---|---|---|
| Product: | [Fedora] Fedora | Reporter: | Andrew Vagin <avagin> |
| Component: | kernel | Assignee: | Kernel Maintainer List <kernel-maint> |
| Status: | CLOSED ERRATA | QA Contact: | Fedora Extras Quality Assurance <extras-qa> |
| Severity: | unspecified | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 24 | CC: | gansalmon, itamar, jonathan, kernel-maint, madhu.chinakonda, mchehab, ptikhomirov |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2016-07-14 11:18:22 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
The problem was triggered in our criu tests, I so I discovered the real problem of it all.
In Linux v4.3 commit df2cf4a78e48 ("IGMP: Inhibit reports for local multicast groups") sysctl igmp_link_local_mcast_reports was introduced in ipv4_net_table.
And in ipv4_net_table it's data was initialized to point on sysctl_igmp_llm_reports variable. That was so before commit 87a8a2ae65b7 ("igmp: Namespaceify igmp_llm_reports sysctl knob").
So next it's data pointer is shifted to the offset of current netnamespace relative to init_net in ipv4_sysctl_init_net function. But that is completely wrong if variable is not net-namespaced, so we get random kernel address and can write/read to/from it one int, that can lead to memory corruption and crashes in random places in kernel.
So conclusion is: we can not touch /proc/sys/net/ipv4/igmp_link_local_mcast_reports in v4.3-v4.5 between those two patches.
Simple reproduction(kernel-4.5.7-202.fc23.x86_64):
while :; do unshare -n sysctl -w net.ipv4.igmp_link_local_mcast_reports=268435456; done
So if 87a8a2ae65b7 is the fix, then the kernel in updates-testing for F24 shouldn't have this problem. Can someone test and confirm? > Can someone test and confirm?
Checked on my F24:
[root@localhost ~]# uname -r
4.6.3-300.fc24.x86_64
I ran these night:
[root@localhost ~]# time while :; do unshare -n sysctl -w net.ipv4.igmp_link_local_mcast_reports=268435456; done
net.ipv4.igmp_link_local_mcast_reports = 268435456
... (for 357846 times)
And everything is OK: no crash, no panic and no warnings.
Great, thank you for testing and the great analysis. Much appreciated. |
Description of problem: cmdline: BOOT_IMAGE=/vmlinuz-4.5.4-300.fc24.x86_64 root=/dev/mapper/fedora-root ro rd.lvm.lv=fedora/swap rd.lvm.lv=fedora/root debug console=ttyS0,115200 console=tty0 crashkernel=128M LANG=en_US.UTF-8 abrt_version: 2.8.0 hostname: zdtm.openvz.org kernel: 4.5.4-300.fc24.x86_64 last_occurrence: 1467417743 pkg_arch: x86_64 pkg_epoch: 0 pkg_name: kernel-core pkg_release: 300.fc24 pkg_version: 4.5.4 runlevel: N 3 backtrace: :BUG: unable to handle kernel NULL pointer dereference at 0000000000000060 :IP: [<ffffffff81264f5d>] copy_tree+0x14d/0x320 :PGD 558e0067 PUD 494bb067 PMD 0 :Oops: 0000 [#1] SMP :Modules linked in: tun nf_conntrack_netlink udp_diag tcp_diag inet_diag netlink_diag af_packet_diag unix_diag binfmt_misc ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_broute bridge stp llc ebtable_filter ebtable_nat ebtables ip6table_raw ip6table_security ip6table_mangle ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_filter ip6_tables iptable_raw iptable_security iptable_mangle iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack crct10dif_pclmul crc32_pclmul ghash_clmulni_intel ppdev sbs sbshc parport_pc parport virtio_balloon acpi_cpufreq shpchp lpc_ich tpm_tis tpm btrfs xor raid6_pq loop nfsd auth_rpcgss nfs_acl lockd grace sunrpc xfs libcrc32c crc32c_intel ata_generic serio_raw pata_acpi e1000 virtio_pci : virtio virtio_ring fjes :CPU: 0 PID: 28656 Comm: criu Not tainted 4.5.4-300.fc24.x86_64 #1 :Hardware name: Parallels Software International Inc. Parallels Virtual Platform/Parallels Virtual Platform, BIOS 6.10.24198.1226784 12/09/2015 :task: ffff880023848000 ti: ffff880026fdc000 task.ti: ffff880026fdc000 :RIP: 0010:[<ffffffff81264f5d>] [<ffffffff81264f5d>] copy_tree+0x14d/0x320 :RSP: 0018:ffff880026fdfcf8 EFLAGS: 00010246 :RAX: 0000000000000000 RBX: ffff880049595200 RCX: ffff880023c2f480 :RDX: 0000000000000004 RSI: 0000000000000000 RDI: ffff880049595260 :RBP: ffff880026fdfd50 R08: ffff8800495947d0 R09: 0000000000000000 :R10: ffffe8ffffddc5d8 R11: 000000000001c5d1 R12: ffff880023c2df80 :R13: ffff88006f3a74ff R14: ffff880049595e00 R15: ffff880049595200 :FS: 00007fca9e5c7800(0000) GS:ffff88007ce00000(0000) knlGS:0000000000000000 :CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 :CR2: 0000000000000060 CR3: 00000000210e2000 CR4: 00000000000406f0 :DR0: 00007f2f1db87030 DR1: 0000000000000000 DR2: 0000000000000000 :DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000600 :Stack: : ffff880023c2e450 ffff880023c2e400 ffff88007c408000 ffff880049595e88 : 00000000de7f9542 0000000400000005 ffff880023c2e400 ffff880026f4fb40 : ffff8800239b3f00 ffffffff81c44da0 ffff8800495bf000 ffff880026fdfd90 :Call Trace: : [<ffffffff81266406>] copy_mnt_ns+0x86/0x280 : [<ffffffff810c5476>] ? create_new_namespaces+0x36/0x1b0 : [<ffffffff810c54a1>] create_new_namespaces+0x61/0x1b0 : [<ffffffff810c565d>] copy_namespaces+0x6d/0xa0 : [<ffffffff810a40f5>] copy_process.part.27+0x805/0x1810 : [<ffffffff810a52bf>] _do_fork+0xcf/0x390 : [<ffffffff810a5629>] SyS_clone+0x19/0x20 : [<ffffffff817d086e>] entry_SYSCALL_64_fastpath+0x12/0x6d :Code: 60 48 39 45 a8 4c 8d 60 a0 75 82 4c 89 f3 48 83 c4 30 48 89 d8 5b 41 5c 41 5d 41 5e 41 5f 5d c3 8b 45 cc 49 8b 75 20 85 c0 75 0e <48> 81 7e 60 80 8f 82 81 0f 84 60 01 00 00 49 8b 45 10 48 39 c1 :RIP [<ffffffff81264f5d>] copy_tree+0x14d/0x320 : RSP <ffff880026fdfcf8> :CR2: 0000000000000060 Here is a place where it crashed: static bool is_mnt_ns_file(struct dentry *dentry) { /* Is this a proxy for a mount namespace? */ return dentry->d_op == &ns_dentry_operations && ffffffff81264f5d: 48 81 7e 60 80 8f 82 cmpq $0xffffffff81828f80,0x60(%rsi) ffffffff81264f64: 81 ffffffff81264f65: 0f 84 60 01 00 00 je ffffffff812650cb <copy_tree+0x2bb> if (!(flag & CL_COPY_MNT_NS_FILE) && is_mnt_ns_file(s->mnt.mnt_root)) { s = skip_mnt_tree(s); continue;