Bug 1539540
Summary: | Kernel panic in overlayfs readdir in Project Atomic | ||
---|---|---|---|
Product: | [Fedora] Fedora | Reporter: | Kenny Root <kenny> |
Component: | kernel | Assignee: | Kernel Maintainer List <kernel-maint> |
Status: | CLOSED INSUFFICIENT_DATA | QA Contact: | Fedora Extras Quality Assurance <extras-qa> |
Severity: | unspecified | Docs Contact: | |
Priority: | unspecified | ||
Version: | 27 | CC: | airlied, bskeggs, ewk, hdegoede, ichavero, itamar, jarodwilson, jglisse, john.j5live, jonathan, josef, kenny, kernel-maint, linville, mchehab, mjg59, mszeredi, steved |
Target Milestone: | --- | Flags: | jforbes:
needinfo?
|
Target Release: | --- | ||
Hardware: | Unspecified | ||
OS: | Linux | ||
Whiteboard: | |||
Fixed In Version: | Doc Type: | If docs needed, set a value | |
Doc Text: | Story Points: | --- | |
Clone Of: | Environment: | ||
Last Closed: | 2018-08-29 15:23:53 UTC | Type: | Bug |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: |
Description
Kenny Root
2018-01-29 07:22:43 UTC
We apologize for the inconvenience. There is a large number of bugs to go through and several of them have gone stale. As kernel maintainers, we try to keep up with bugzilla but due the rate at which the upstream kernel project moves, bugs may be fixed without any indication to us. Due to this, we are doing a mass bug update across all of the Fedora 27 kernel bugs. Fedora 27 has now been rebased to 4.15.3-300.f27. Please test this kernel update (or newer) and let us know if you issue has been resolved or if it is still present with the newer kernel. If you experience different issues, please open a new bug report for those. Yes, this still happens with 4.15.4-300.fc27 with almost the same stack: [82732.045016] WARNING: CPU: 3 PID: 22698 at fs/overlayfs/readdir.c:411 ovl_iterate+0x254/0x260 [overlay] [82732.049215] Modules linked in: veth xt_nat ipt_MASQUERADE nf_nat_masquerade_ipv4 overlay ip6table_nat nf_nat_ipv6 ip6t_REJECT nf_reject_ipv6 ip6table_mangle ip6table_raw nf_conntrack_ipv6 nf_defrag_ipv6 nf_log_ipv6 ip6table_filter ip6_tables xt_recent iptable_nat nf_nat_ipv4 xt_comment xt_physdev br_netfilter bridge stp llc xt_mark iptable_mangle xt_addrtype xt_CT iptable_raw xt_multiport nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack xt_NFLOG nfnetlink_log nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_sane nf_conntrack_netlink nfnetlink nf_nat_tftp nf_nat_snmp_basic nf_conntrack_snmp nf_nat_sip nf_nat_pptp nf_nat_proto_gre nf_nat_irc nf_nat_h323 nf_nat_ftp nf_nat_amanda nf_nat nf_conntrack_tftp nf_conntrack_sip nf_conntrack_pptp nf_conntrack_proto_gre nf_conntrack_netbios_ns nf_conntrack_broadcast [82732.073383] nf_conntrack_irc nf_conntrack_h323 nf_conntrack_ftp ts_kmp nf_conntrack_amanda nf_conntrack nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache snd_hda_codec_generic snd_hda_intel snd_hda_codec snd_pcsp qxl snd_hda_core ttm snd_hwdep snd_pcm drm_kms_helper snd_timer snd soundcore drm i2c_piix4 joydev virtio_balloon qemu_fw_cfg virtio_console virtio_rng xfs libcrc32c virtio_net virtio_blk crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel virtio_pci ata_generic virtio_ring serio_raw virtio pata_acpi sunrpc scsi_transport_iscsi [82732.090079] CPU: 3 PID: 22698 Comm: chown Tainted: G W 4.15.4-300.fc27.x86_64 #1 [82732.093152] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.10.2-2.fc27 04/01/2014 [82732.096217] RIP: 0010:ovl_iterate+0x254/0x260 [overlay] [82732.098709] RSP: 0018:ffffb58082427e40 EFLAGS: 00010246 [82732.101131] RAX: 0000000000000000 RBX: ffff9cd72e74b780 RCX: ffff9cd8347f7b20 [82732.103870] RDX: ffffffff00000001 RSI: ffff9cd75cd41040 RDI: ffff9cd72e74b780 [82732.106588] RBP: ffffb58082427ec0 R08: 0000000000000000 R09: 0000000000000000 [82732.109274] R10: ffffb58082427f38 R11: 0000000000000000 R12: ffff9cd810352340 [82732.111980] R13: ffff9cd77346ea00 R14: ffff9cd838f8f200 R15: 0000000000000000 [82732.114642] FS: 00007faaec842740(0000) GS:ffff9cd83fd80000(0000) knlGS:0000000000000000 [82732.117555] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [82732.120064] CR2: 0000000000b71288 CR3: 00000000637b0002 CR4: 00000000001606e0 [82732.122798] Call Trace: [82732.124665] iterate_dir+0x166/0x180 [82732.126680] SyS_getdents+0xa0/0x130 [82732.128635] ? fillonedir+0xe0/0xe0 [82732.130587] ? do_syscall_64+0x75/0x180 [82732.132536] do_syscall_64+0x75/0x180 [82732.134417] entry_SYSCALL_64_after_hwframe+0x21/0x86 [82732.136620] RIP: 0033:0x7faaec3263b5 [82732.138451] RSP: 002b:00007ffd556b7730 EFLAGS: 00000246 ORIG_RAX: 000000000000004e [82732.140941] RAX: ffffffffffffffda RBX: 0000000000b69280 RCX: 00007faaec3263b5 [82732.143319] RDX: 0000000000008000 RSI: 0000000000b69280 RDI: 0000000000000003 [82732.145760] RBP: 0000000000b69280 R08: 0000000000000000 R09: 00007faaec6277b8 [82732.148287] R10: 00007ffd556b7550 R11: 0000000000000246 R12: ffffffffffffff60 [82732.150745] R13: 0000000000000000 R14: 00000000ffffff01 R15: 0000000000000000 [82732.153213] Code: fe ff ff 4c 89 ff e8 6c f8 ff ff 4c 89 f7 e8 f4 6a 8e c5 4c 63 74 24 04 eb a6 49 8b 06 48 85 c0 74 09 48 83 c0 01 49 89 06 eb 95 <0f> ff eb f3 44 89 f0 e9 4e fe ff ff 0f 1f 44 00 00 41 57 41 56 [82732.158959] ---[ end trace b4a813a724f1649c ]--- [82732.161864] WARNING: CPU: 3 PID: 22698 at fs/overlayfs/readdir.c:411 ovl_iterate+0x254/0x260 [overlay] [82732.164627] Modules linked in: veth xt_nat ipt_MASQUERADE nf_nat_masquerade_ipv4 overlay ip6table_nat nf_nat_ipv6 ip6t_REJECT nf_reject_ipv6 ip6table_mangle ip6table_raw nf_conntrack_ipv6 nf_defrag_ipv6 nf_log_ipv6 ip6table_filter ip6_tables xt_recent iptable_nat nf_nat_ipv4 xt_comment xt_physdev br_netfilter bridge stp llc xt_mark iptable_mangle xt_addrtype xt_CT iptable_raw xt_multiport nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack xt_NFLOG nfnetlink_log nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_sane nf_conntrack_netlink nfnetlink nf_nat_tftp nf_nat_snmp_basic nf_conntrack_snmp nf_nat_sip nf_nat_pptp nf_nat_proto_gre nf_nat_irc nf_nat_h323 nf_nat_ftp nf_nat_amanda nf_nat nf_conntrack_tftp nf_conntrack_sip nf_conntrack_pptp nf_conntrack_proto_gre nf_conntrack_netbios_ns nf_conntrack_broadcast [82732.185412] nf_conntrack_irc nf_conntrack_h323 nf_conntrack_ftp ts_kmp nf_conntrack_amanda nf_conntrack nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache snd_hda_codec_generic snd_hda_intel snd_hda_codec snd_pcsp qxl snd_hda_core ttm snd_hwdep snd_pcm drm_kms_helper snd_timer snd soundcore drm i2c_piix4 joydev virtio_balloon qemu_fw_cfg virtio_coa_generic virtio_ring serio_raw virtio pata_acpi sunrpc scsi_transport_iscsi [82732.200956] CPU: 3 PID: 22698 Comm: chown Tainted: G W 4.15.4-300.fc27.x86_64 #1 [82732.203674] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.10.2-2.fc27 04/01/2014 [82732.206404] RIP: 0010:ovl_iterate+0x254/0x260 [overlay] [82732.208525] RSP: 0018:ffffb58082427e40 EFLAGS: 00010246 [82732.210622] RAX: 0000000000000000 RBX: ffff9cd763075000 RCX: ffff9cd8347f7b20 [82732.213002] RDX: ffffffff00000001 RSI: ffff9cd75cd40020 RDI: ffff9cd763075000 [82732.215413] RBP: ffffb58082427ec0 R08: 0000000000000000 R09: 0000000000000000 [82732.217787] R10: ffffb58082427f38 R11: 0000000000000000 R12: ffff9cd810352e00 [82732.220160] R13: ffff9cd77346e800 R14: ffff9cd838f8f540 R15: 0000000000000000 [82732.222567] FS: 00007faaec842740(0000) GS:ffff9cd83fd80000(0000) knlGS:0000000000000000 [82732.225093] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [82732.227221] CR2: 0000000000b71288 CR3: 00000000637b0002 CR4: 00000000001606e0 [82732.229598] Call Trace: [82732.231180] iterate_dir+0x166/0x180 [82732.232999] SyS_getdents+0xa0/0x130 [82732.234989] ? fillonedir+0xe0/0xe0 [82732.236861] ? do_syscall_64+0x75/0x180 [82732.238885] do_syscall_64+0x75/0x180 [82732.240774] entry_SYSCALL_64_after_hwframe+0x21/0x86 [82732.243005] RIP: 0033:0x7faaec3263b5 [82732.244895] RSP: 002b:00007ffd556b7730 EFLAGS: 00000246 ORIG_RAX: 000000000000004e [82732.247659] RAX: ffffffffffffffda RBX: 0000000000b69280 RCX: 00007faaec3263b5 [82732.250345] RDX: 0000000000008000 RSI: 0000000000b69280 RDI: 0000000000000003 [82732.253054] RBP: 0000000000b69280 R08: 0000000000000001 R09: 0000000000000076 [82732.255754] R10: 0000000000000000 R11: 0000000000000246 R12: ffffffffffffff60 [82732.258473] R13: 0000000000000000 R14: 00000000ffffff01 R15: 0000000000000000 [82732.261178] Code: fe ff ff 4c 89 ff e8 6c f8 ff ff 4c 89 f7 e8 f4 6a 8e c5 4c 63 74 24 04 eb a6 49 8b 06 48 85 c0 74 09 48 83 c0 01 49 89 06 eb 95 <0f> ff eb f3 44 89 f0 e9 4e fe ff ff 0f 1f 44 00 00 41 57 41 56 [82732.267670] ---[ end trace b4a813a724f1649d ]--- [82732.269942] WARNING: CPU: 0 PID: 22698 at fs/overlayfs/readdir.c:411 ovl_iterate+0x254/0x260 [overlay] [82732.275449] Modules linked in: veth xt_nat ipt_MASQUERADE nf_nat_masquerade_ipv4 overlay ip6table_nat nf_nat_ipv6 ip6t_REJECT nf_reject_ipv6 ip6table_mangle ip6table_raw nf_conntrack_ipv6 nf_defrag_ipv6 nf_log_ipv6 ip6table_filter ip6_tables xt_recent iptable_nat nf_nat_ipv4 xt_comment xt_physdev br_netfilter bridge stp llc xt_mark iptable_mangle xt_addrtype xt_CT iptable_raw xt_multiport nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack xt_NFLOG nfnetlink_log nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_sane nf_conntrack_netlink nfnetlink nf_nat_tftp nf_nat_snmp_basic nf_conntrack_snmp nf_nat_sip nf_nat_pptp nf_nat_proto_gre nf_nat_irc nf_nat_h323 nf_nat_ftp nf_nat_amanda nf_nat nf_conntrack_tftp nf_conntrack_sip nf_conntrack_pptp nf_conntrack_proto_gre nf_conntrack_netbios_ns nf_conntrack_broadcast [82732.299415] nf_conntrack_irc nf_conntrack_h323 nf_conntrack_ftp ts_kmp nf_conntrack_amanda nf_conntrack nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache snd_hda_codec_generic snd_hda_intel snd_hda_codec snd_pcsp qxl snd_hda_core ttm snd_hwdep snd_pcm drm_kms_helper snd_timer snd soundcore drm i2c_piix4 joydev virtio_balloon qemu_fw_cfg virtio_console virtio_rng xfs libcrc32c virtio_net virtio_blk crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel virtio_pci ata_generic virtio_ring serio_raw virtio pata_acpi sunrpc scsi_transport_iscsi [82732.316443] CPU: 0 PID: 22698 Comm: chown Tainted: G W 4.15.4-300.fc27.x86_64 #1 [82732.319585] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.10.2-2.fc27 04/01/2014 [82732.322800] RIP: 0010:ovl_iterate+0x254/0x260 [overlay] [82732.325141] RSP: 0018:ffffb58082427e40 EFLAGS: 00010246 [82732.327482] RAX: 0000000000000000 RBX: ffff9cd71dc75000 RCX: ffff9cd8347f7b20 [82732.330229] RDX: ffffffff00000001 RSI: ffff9cd75cd43330 RDI: ffff9cd71dc75000 [82732.332888] RBP: ffffb58082427ec0 R08: 0000000000000000 R09: 0000000000000000 [82732.335635] R10: ffffb58082427f38 R11: 0000000000000000 R12: ffff9cd81ba1ff40 [82732.338387] R13: ffff9cd75adde300 R14: ffff9cd838f8f100 R15: 0000000000000000 [82732.341143] FS: 00007faaec842740(0000) GS:ffff9cd83fc00000(0000) knlGS:0000000000000000 [82732.344064] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [82732.346433] CR2: 000000c420189000 CR3: 00000000637b0001 CR4: 00000000001606f0 [82732.349073] Call Trace: [82732.350758] iterate_dir+0x166/0x180 [82732.352640] SyS_getdents+0xa0/0x130 [82732.354547] ? fillonedir+0xe0/0xe0 [82732.356391] ? do_syscall_64+0x75/0x180 [82732.358363] do_syscall_64+0x75/0x180 [82732.360206] entry_SYSCALL_64_after_hwframe+0x21/0x86 [82732.362341] RIP: 0033:0x7faaec3263b5 [82732.364174] RSP: 002b:00007ffd556b7730 EFLAGS: 00000246 ORIG_RAX: 000000000000004e [82732.366803] RAX: ffffffffffffffda RBX: 0000000000b69280 RCX: 00007faaec3263b5 [82732.369371] RDX: 0000000000008000 RSI: 0000000000b69280 RDI: 0000000000000003 [82732.371938] RBP: 0000000000b69280 R08: 0000000000000001 R09: 00007faaec6277b8 [82732.374462] R10: 0000000000000000 R11: 0000000000000246 R12: ffffffffffffff60 [82732.377030] R13: 0000000000000000 R14: 00000000ffffff01 R15: 0000000000000000 [82732.379594] Code: fe ff ff 4c 89 ff e8 6c f8 ff ff 4c 89 f7 e8 f4 6a 8e c5 4c 63 74 24 04 eb a6 49 8b 06 48 85 c0 74 09 48 83 c0 01 49 89 06 eb 95 <0f> ff eb f3 44 89 f0 e9 4e fe ff ff 0f 1f 44 00 00 41 57 41 56 [82732.385675] ---[ end trace b4a813a724f1649e ]--- [82732.593364] systemd-journald[18005]: Received SIGTERM from PID 22615 (my_init). [82822.825588] kauditd_printk_skb: 108 callbacks suppressed It's possibly this line that is triggering the bug: https://github.com/linuxserver/dockergui/blob/f1047818b6c27235d255fb22ac7ea837f0371d76/files/install/install.sh#L84 Repro case on Project Atomic 27.93: docker run --rm coppit/filebot Output to console before panic happens: *** Running /etc/my_init.d/01_user_config.sh... groupmod: GID '0' already exists usermod: UID '0' already exists usermod: no changes [ 625.293290] WARNING: CPU: 1 PID: 10221 at fs/overlayfs/readdir.c:411 ovl_iterate+0x254/0x260 [overlay] The error might be caused by this if statement not passing https://elixir.bootlin.com/linux/v4.15.4/source/fs/overlayfs/readdir.c#L244 ovl_dir_release calls ovl_cache_put with od->cache and ovl_dir_cache(d_inode(dentry)) cache not matching which causes the refcount to be lowered to zero, but not removed. if (!cache->refcount) { if (ovl_dir_cache(d_inode(dentry)) == cache) <---- HERE ovl_set_dir_cache(d_inode(dentry), NULL); ovl_cache_free(&cache->entries); kfree(cache); } I'm not sure how they get out of sync. Adding Miklos Szeredi who added the WARN_ON that is triggering in 4.14: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=4edb83bb1041e2f946ce36ea93f6bcd06d984bf4 *********** MASS BUG UPDATE ************** We apologize for the inconvenience. There are a large number of bugs to go through and several of them have gone stale. Due to this, we are doing a mass bug update across all of the Fedora 27 kernel bugs. Fedora 27 has now been rebased to 4.17.7-100.fc27. Please test this kernel update (or newer) and let us know if you issue has been resolved or if it is still present with the newer kernel. If you have moved on to Fedora 28, and are still experiencing this issue, please change the version to Fedora 28. If you experience different issues, please open a new bug report for those. *********** MASS BUG UPDATE ************** This bug is being closed with INSUFFICIENT_DATA as there has not been a response in 5 weeks. If you are still experiencing this issue, please reopen and attach the relevant data from the latest kernel you are running and any data that might have been requested previously. |