Description of problem: When starting up containers (docker autostart) on Project Atomic, the kernel will panic when it tries to call chown on some file Version-Release number of selected component (if applicable): 4.14.13-300.fc27.x86_64 How reproducible: Currently the filesystem is in such a state that this is 100% reproducible on this device. Steps to Reproduce: 1. Create about 12 docker images 2. Set them to autostart 3. Actual results: Kernel panic upon reboot Expected results: Device allows disk access without kernel panic Additional info: I didn't try to get a reproduction from scratch, but here is a kernel panic: [ 150.815546] ------------[ cut here ]------------ [ 150.819139] WARNING: CPU: 3 PID: 9464 at fs/overlayfs/readdir.c:393 ovl_iterate+0x255/0x270 [overlay] [ 150.822723] Modules linked in: veth xt_nat ipt_MASQUERADE nf_nat_masquerade_ipv4 overlay iptable_nat nf_nat_ipv4 iptable_mangle iptable_raw nf_conntrack_ipv4 nf_defrag_ipv4 xt_recent ip6table_nat nf_nat_ipv6 xt_comment ip6t_REJECT nf_reject_ipv6 xt_addrtype x t_physdev br_netfilter bridge stp llc xt_mark ip6table_mangle xt_CT ip6table_raw xt_multiport nf_log_ipv4 nf_conntrack_ipv6 nf_nat_ tftp nf_nat_snmp_basic nf_defrag_ipv6 nf_conntrack_snmp nf_nat_sip xt_conntrack nf_nat_pptp nf_nat_proto_gre nf_nat_irc nf_nat_h323 xt_NFLOG nfnetlink_log nf_nat_ftp nf_nat_amanda xt_LOG nf_nat nf_log_ipv6 nf_log_common nf_conntrack_tftp nf_conntrack_sip nf_conn track_sane nf_conntrack_pptp nf_conntrack_proto_gre nf_conntrack_netlink nfnetlink nf_conntrack_netbios_ns nf_conntrack_broadcast n f_conntrack_irc nf_conntrack_h323 [ 150.847332] br-0ab0b5690df4: port 1(veth5eff4e6) entered blocking state [ 150.847334] br-0ab0b5690df4: port 1(veth5eff4e6) entered disabled state [ 150.847388] device veth5eff4e6 entered promiscuous mode [ 150.848196] IPv6: ADDRCONF(NETDEV_UP): veth5eff4e6: link is not ready [ 150.861393] ts_kmp nf_conntrack_amanda nf_conntrack_ftp nf_conntrack nfsv3 nfs_acl rpcsec_gss_krb5 ip6table_filter ip6_tables a uth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache qxl snd_hda_codec_generic snd_hda_intel ttm drm_kms_helper drm snd_hda_codec ppdev snd_hda_core snd_hwdep snd_pcsp snd_pcm snd_timer snd i2c_piix4 parport_pc soundcore joydev virtio_balloon parport xfs libcrc 32c virtio_console virtio_net virtio_blk crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel serio_raw virtio_pci ata_ge neric pata_acpi qemu_fw_cfg sunrpc scsi_transport_iscsi virtio_rng virtio_ring virtio [ 150.880767] CPU: 3 PID: 9464 Comm: chown Tainted: G W 4.14.13-300.fc27.x86_64 #1 [ 150.884418] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.10.2-2.fc27 04/01/2014 [ 150.887795] task: ffff8fd71fc73e80 task.stack: ffffbaa3c1c78000 [ 150.890511] RIP: 0010:ovl_iterate+0x255/0x270 [overlay] [ 150.893038] RSP: 0018:ffffbaa3c1c7be28 EFLAGS: 00010246 [ 150.895550] RAX: 0000000000000000 RBX: ffff8fd73915a300 RCX: 0000000000000001 [ 150.898509] RDX: ffffffff00000001 RSI: ffff8fd72b2065c0 RDI: ffff8fd73915a300 [ 150.901444] RBP: ffffbaa3c1c7be60 R08: 0000000000000000 R09: 0000000000000000 [ 150.904764] R10: ffffbaa3c1c7bf30 R11: 0000000000000000 R12: ffffbaa3c1c7beb0 [ 150.907675] R13: ffff8fd72b2065c0 R14: ffff8fd7271b8900 R15: ffff8fd7270b0680 [ 150.910561] FS: 00007f98c2495740(0000) GS:ffff8fd73fd80000(0000) knlGS:0000000000000000 [ 150.913782] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 150.916310] CR2: 0000000000e4d288 CR3: 000000012b148004 CR4: 00000000001606e0 [ 150.919135] Call Trace: [ 150.920870] iterate_dir+0xce/0x1a0 [ 150.922811] SyS_getdents+0xaa/0x140 [ 150.924723] ? compat_fillonedir+0x110/0x110 [ 150.926813] do_syscall_64+0x67/0x180 [ 150.928762] ? do_syscall_64+0x67/0x180 [ 150.930726] entry_SYSCALL64_slow_path+0x25/0x25 [ 150.934234] RIP: 0033:0x7f98c1f793b5 [ 150.938772] RSP: 002b:00007ffdad2b5e90 EFLAGS: 00000246 ORIG_RAX: 000000000000004e [ 150.942004] RAX: ffffffffffffffda RBX: 0000000000e45280 RCX: 00007f98c1f793b5 [ 150.944387] RDX: 0000000000008000 RSI: 0000000000e45280 RDI: 0000000000000003 [ 150.946666] RBP: 0000000000e45280 R08: 0000000000000000 R09: 00007f98c227a7b8 [ 150.948817] R10: 00007ffdad2b5cb0 R11: 0000000000000246 R12: ffffffffffffff60 [ 150.950995] R13: 0000000000000000 R14: 00000000ffffff01 R15: 0000000000000000 [ 150.953162] Code: 48 8b 4d c8 48 89 cf e8 5a f8 ff ff 4c 89 ff e8 f2 d3 9f dc 4c 63 7d d4 eb a4 49 8b 07 48 85 c0 74 09 48 83 c0 01 49 89 07 eb 93 <0f> ff eb f3 44 89 f8 e9 48 fe ff ff 0f 1f 44 00 00 66 2e 0f 1f [ 150.958249] ---[ end trace 377f9f46c7c15305 ]--- [ 150.961226] ------------[ cut here ]------------
We apologize for the inconvenience. There is a large number of bugs to go through and several of them have gone stale. As kernel maintainers, we try to keep up with bugzilla but due the rate at which the upstream kernel project moves, bugs may be fixed without any indication to us. Due to this, we are doing a mass bug update across all of the Fedora 27 kernel bugs. Fedora 27 has now been rebased to 4.15.3-300.f27. Please test this kernel update (or newer) and let us know if you issue has been resolved or if it is still present with the newer kernel. If you experience different issues, please open a new bug report for those.
Yes, this still happens with 4.15.4-300.fc27 with almost the same stack: [82732.045016] WARNING: CPU: 3 PID: 22698 at fs/overlayfs/readdir.c:411 ovl_iterate+0x254/0x260 [overlay] [82732.049215] Modules linked in: veth xt_nat ipt_MASQUERADE nf_nat_masquerade_ipv4 overlay ip6table_nat nf_nat_ipv6 ip6t_REJECT nf_reject_ipv6 ip6table_mangle ip6table_raw nf_conntrack_ipv6 nf_defrag_ipv6 nf_log_ipv6 ip6table_filter ip6_tables xt_recent iptable_nat nf_nat_ipv4 xt_comment xt_physdev br_netfilter bridge stp llc xt_mark iptable_mangle xt_addrtype xt_CT iptable_raw xt_multiport nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack xt_NFLOG nfnetlink_log nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_sane nf_conntrack_netlink nfnetlink nf_nat_tftp nf_nat_snmp_basic nf_conntrack_snmp nf_nat_sip nf_nat_pptp nf_nat_proto_gre nf_nat_irc nf_nat_h323 nf_nat_ftp nf_nat_amanda nf_nat nf_conntrack_tftp nf_conntrack_sip nf_conntrack_pptp nf_conntrack_proto_gre nf_conntrack_netbios_ns nf_conntrack_broadcast [82732.073383] nf_conntrack_irc nf_conntrack_h323 nf_conntrack_ftp ts_kmp nf_conntrack_amanda nf_conntrack nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache snd_hda_codec_generic snd_hda_intel snd_hda_codec snd_pcsp qxl snd_hda_core ttm snd_hwdep snd_pcm drm_kms_helper snd_timer snd soundcore drm i2c_piix4 joydev virtio_balloon qemu_fw_cfg virtio_console virtio_rng xfs libcrc32c virtio_net virtio_blk crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel virtio_pci ata_generic virtio_ring serio_raw virtio pata_acpi sunrpc scsi_transport_iscsi [82732.090079] CPU: 3 PID: 22698 Comm: chown Tainted: G W 4.15.4-300.fc27.x86_64 #1 [82732.093152] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.10.2-2.fc27 04/01/2014 [82732.096217] RIP: 0010:ovl_iterate+0x254/0x260 [overlay] [82732.098709] RSP: 0018:ffffb58082427e40 EFLAGS: 00010246 [82732.101131] RAX: 0000000000000000 RBX: ffff9cd72e74b780 RCX: ffff9cd8347f7b20 [82732.103870] RDX: ffffffff00000001 RSI: ffff9cd75cd41040 RDI: ffff9cd72e74b780 [82732.106588] RBP: ffffb58082427ec0 R08: 0000000000000000 R09: 0000000000000000 [82732.109274] R10: ffffb58082427f38 R11: 0000000000000000 R12: ffff9cd810352340 [82732.111980] R13: ffff9cd77346ea00 R14: ffff9cd838f8f200 R15: 0000000000000000 [82732.114642] FS: 00007faaec842740(0000) GS:ffff9cd83fd80000(0000) knlGS:0000000000000000 [82732.117555] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [82732.120064] CR2: 0000000000b71288 CR3: 00000000637b0002 CR4: 00000000001606e0 [82732.122798] Call Trace: [82732.124665] iterate_dir+0x166/0x180 [82732.126680] SyS_getdents+0xa0/0x130 [82732.128635] ? fillonedir+0xe0/0xe0 [82732.130587] ? do_syscall_64+0x75/0x180 [82732.132536] do_syscall_64+0x75/0x180 [82732.134417] entry_SYSCALL_64_after_hwframe+0x21/0x86 [82732.136620] RIP: 0033:0x7faaec3263b5 [82732.138451] RSP: 002b:00007ffd556b7730 EFLAGS: 00000246 ORIG_RAX: 000000000000004e [82732.140941] RAX: ffffffffffffffda RBX: 0000000000b69280 RCX: 00007faaec3263b5 [82732.143319] RDX: 0000000000008000 RSI: 0000000000b69280 RDI: 0000000000000003 [82732.145760] RBP: 0000000000b69280 R08: 0000000000000000 R09: 00007faaec6277b8 [82732.148287] R10: 00007ffd556b7550 R11: 0000000000000246 R12: ffffffffffffff60 [82732.150745] R13: 0000000000000000 R14: 00000000ffffff01 R15: 0000000000000000 [82732.153213] Code: fe ff ff 4c 89 ff e8 6c f8 ff ff 4c 89 f7 e8 f4 6a 8e c5 4c 63 74 24 04 eb a6 49 8b 06 48 85 c0 74 09 48 83 c0 01 49 89 06 eb 95 <0f> ff eb f3 44 89 f0 e9 4e fe ff ff 0f 1f 44 00 00 41 57 41 56 [82732.158959] ---[ end trace b4a813a724f1649c ]--- [82732.161864] WARNING: CPU: 3 PID: 22698 at fs/overlayfs/readdir.c:411 ovl_iterate+0x254/0x260 [overlay] [82732.164627] Modules linked in: veth xt_nat ipt_MASQUERADE nf_nat_masquerade_ipv4 overlay ip6table_nat nf_nat_ipv6 ip6t_REJECT nf_reject_ipv6 ip6table_mangle ip6table_raw nf_conntrack_ipv6 nf_defrag_ipv6 nf_log_ipv6 ip6table_filter ip6_tables xt_recent iptable_nat nf_nat_ipv4 xt_comment xt_physdev br_netfilter bridge stp llc xt_mark iptable_mangle xt_addrtype xt_CT iptable_raw xt_multiport nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack xt_NFLOG nfnetlink_log nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_sane nf_conntrack_netlink nfnetlink nf_nat_tftp nf_nat_snmp_basic nf_conntrack_snmp nf_nat_sip nf_nat_pptp nf_nat_proto_gre nf_nat_irc nf_nat_h323 nf_nat_ftp nf_nat_amanda nf_nat nf_conntrack_tftp nf_conntrack_sip nf_conntrack_pptp nf_conntrack_proto_gre nf_conntrack_netbios_ns nf_conntrack_broadcast [82732.185412] nf_conntrack_irc nf_conntrack_h323 nf_conntrack_ftp ts_kmp nf_conntrack_amanda nf_conntrack nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache snd_hda_codec_generic snd_hda_intel snd_hda_codec snd_pcsp qxl snd_hda_core ttm snd_hwdep snd_pcm drm_kms_helper snd_timer snd soundcore drm i2c_piix4 joydev virtio_balloon qemu_fw_cfg virtio_coa_generic virtio_ring serio_raw virtio pata_acpi sunrpc scsi_transport_iscsi [82732.200956] CPU: 3 PID: 22698 Comm: chown Tainted: G W 4.15.4-300.fc27.x86_64 #1 [82732.203674] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.10.2-2.fc27 04/01/2014 [82732.206404] RIP: 0010:ovl_iterate+0x254/0x260 [overlay] [82732.208525] RSP: 0018:ffffb58082427e40 EFLAGS: 00010246 [82732.210622] RAX: 0000000000000000 RBX: ffff9cd763075000 RCX: ffff9cd8347f7b20 [82732.213002] RDX: ffffffff00000001 RSI: ffff9cd75cd40020 RDI: ffff9cd763075000 [82732.215413] RBP: ffffb58082427ec0 R08: 0000000000000000 R09: 0000000000000000 [82732.217787] R10: ffffb58082427f38 R11: 0000000000000000 R12: ffff9cd810352e00 [82732.220160] R13: ffff9cd77346e800 R14: ffff9cd838f8f540 R15: 0000000000000000 [82732.222567] FS: 00007faaec842740(0000) GS:ffff9cd83fd80000(0000) knlGS:0000000000000000 [82732.225093] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [82732.227221] CR2: 0000000000b71288 CR3: 00000000637b0002 CR4: 00000000001606e0 [82732.229598] Call Trace: [82732.231180] iterate_dir+0x166/0x180 [82732.232999] SyS_getdents+0xa0/0x130 [82732.234989] ? fillonedir+0xe0/0xe0 [82732.236861] ? do_syscall_64+0x75/0x180 [82732.238885] do_syscall_64+0x75/0x180 [82732.240774] entry_SYSCALL_64_after_hwframe+0x21/0x86 [82732.243005] RIP: 0033:0x7faaec3263b5 [82732.244895] RSP: 002b:00007ffd556b7730 EFLAGS: 00000246 ORIG_RAX: 000000000000004e [82732.247659] RAX: ffffffffffffffda RBX: 0000000000b69280 RCX: 00007faaec3263b5 [82732.250345] RDX: 0000000000008000 RSI: 0000000000b69280 RDI: 0000000000000003 [82732.253054] RBP: 0000000000b69280 R08: 0000000000000001 R09: 0000000000000076 [82732.255754] R10: 0000000000000000 R11: 0000000000000246 R12: ffffffffffffff60 [82732.258473] R13: 0000000000000000 R14: 00000000ffffff01 R15: 0000000000000000 [82732.261178] Code: fe ff ff 4c 89 ff e8 6c f8 ff ff 4c 89 f7 e8 f4 6a 8e c5 4c 63 74 24 04 eb a6 49 8b 06 48 85 c0 74 09 48 83 c0 01 49 89 06 eb 95 <0f> ff eb f3 44 89 f0 e9 4e fe ff ff 0f 1f 44 00 00 41 57 41 56 [82732.267670] ---[ end trace b4a813a724f1649d ]--- [82732.269942] WARNING: CPU: 0 PID: 22698 at fs/overlayfs/readdir.c:411 ovl_iterate+0x254/0x260 [overlay] [82732.275449] Modules linked in: veth xt_nat ipt_MASQUERADE nf_nat_masquerade_ipv4 overlay ip6table_nat nf_nat_ipv6 ip6t_REJECT nf_reject_ipv6 ip6table_mangle ip6table_raw nf_conntrack_ipv6 nf_defrag_ipv6 nf_log_ipv6 ip6table_filter ip6_tables xt_recent iptable_nat nf_nat_ipv4 xt_comment xt_physdev br_netfilter bridge stp llc xt_mark iptable_mangle xt_addrtype xt_CT iptable_raw xt_multiport nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack xt_NFLOG nfnetlink_log nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_sane nf_conntrack_netlink nfnetlink nf_nat_tftp nf_nat_snmp_basic nf_conntrack_snmp nf_nat_sip nf_nat_pptp nf_nat_proto_gre nf_nat_irc nf_nat_h323 nf_nat_ftp nf_nat_amanda nf_nat nf_conntrack_tftp nf_conntrack_sip nf_conntrack_pptp nf_conntrack_proto_gre nf_conntrack_netbios_ns nf_conntrack_broadcast [82732.299415] nf_conntrack_irc nf_conntrack_h323 nf_conntrack_ftp ts_kmp nf_conntrack_amanda nf_conntrack nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache snd_hda_codec_generic snd_hda_intel snd_hda_codec snd_pcsp qxl snd_hda_core ttm snd_hwdep snd_pcm drm_kms_helper snd_timer snd soundcore drm i2c_piix4 joydev virtio_balloon qemu_fw_cfg virtio_console virtio_rng xfs libcrc32c virtio_net virtio_blk crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel virtio_pci ata_generic virtio_ring serio_raw virtio pata_acpi sunrpc scsi_transport_iscsi [82732.316443] CPU: 0 PID: 22698 Comm: chown Tainted: G W 4.15.4-300.fc27.x86_64 #1 [82732.319585] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.10.2-2.fc27 04/01/2014 [82732.322800] RIP: 0010:ovl_iterate+0x254/0x260 [overlay] [82732.325141] RSP: 0018:ffffb58082427e40 EFLAGS: 00010246 [82732.327482] RAX: 0000000000000000 RBX: ffff9cd71dc75000 RCX: ffff9cd8347f7b20 [82732.330229] RDX: ffffffff00000001 RSI: ffff9cd75cd43330 RDI: ffff9cd71dc75000 [82732.332888] RBP: ffffb58082427ec0 R08: 0000000000000000 R09: 0000000000000000 [82732.335635] R10: ffffb58082427f38 R11: 0000000000000000 R12: ffff9cd81ba1ff40 [82732.338387] R13: ffff9cd75adde300 R14: ffff9cd838f8f100 R15: 0000000000000000 [82732.341143] FS: 00007faaec842740(0000) GS:ffff9cd83fc00000(0000) knlGS:0000000000000000 [82732.344064] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [82732.346433] CR2: 000000c420189000 CR3: 00000000637b0001 CR4: 00000000001606f0 [82732.349073] Call Trace: [82732.350758] iterate_dir+0x166/0x180 [82732.352640] SyS_getdents+0xa0/0x130 [82732.354547] ? fillonedir+0xe0/0xe0 [82732.356391] ? do_syscall_64+0x75/0x180 [82732.358363] do_syscall_64+0x75/0x180 [82732.360206] entry_SYSCALL_64_after_hwframe+0x21/0x86 [82732.362341] RIP: 0033:0x7faaec3263b5 [82732.364174] RSP: 002b:00007ffd556b7730 EFLAGS: 00000246 ORIG_RAX: 000000000000004e [82732.366803] RAX: ffffffffffffffda RBX: 0000000000b69280 RCX: 00007faaec3263b5 [82732.369371] RDX: 0000000000008000 RSI: 0000000000b69280 RDI: 0000000000000003 [82732.371938] RBP: 0000000000b69280 R08: 0000000000000001 R09: 00007faaec6277b8 [82732.374462] R10: 0000000000000000 R11: 0000000000000246 R12: ffffffffffffff60 [82732.377030] R13: 0000000000000000 R14: 00000000ffffff01 R15: 0000000000000000 [82732.379594] Code: fe ff ff 4c 89 ff e8 6c f8 ff ff 4c 89 f7 e8 f4 6a 8e c5 4c 63 74 24 04 eb a6 49 8b 06 48 85 c0 74 09 48 83 c0 01 49 89 06 eb 95 <0f> ff eb f3 44 89 f0 e9 4e fe ff ff 0f 1f 44 00 00 41 57 41 56 [82732.385675] ---[ end trace b4a813a724f1649e ]--- [82732.593364] systemd-journald[18005]: Received SIGTERM from PID 22615 (my_init). [82822.825588] kauditd_printk_skb: 108 callbacks suppressed
It's possibly this line that is triggering the bug: https://github.com/linuxserver/dockergui/blob/f1047818b6c27235d255fb22ac7ea837f0371d76/files/install/install.sh#L84 Repro case on Project Atomic 27.93: docker run --rm coppit/filebot Output to console before panic happens: *** Running /etc/my_init.d/01_user_config.sh... groupmod: GID '0' already exists usermod: UID '0' already exists usermod: no changes [ 625.293290] WARNING: CPU: 1 PID: 10221 at fs/overlayfs/readdir.c:411 ovl_iterate+0x254/0x260 [overlay]
The error might be caused by this if statement not passing https://elixir.bootlin.com/linux/v4.15.4/source/fs/overlayfs/readdir.c#L244 ovl_dir_release calls ovl_cache_put with od->cache and ovl_dir_cache(d_inode(dentry)) cache not matching which causes the refcount to be lowered to zero, but not removed. if (!cache->refcount) { if (ovl_dir_cache(d_inode(dentry)) == cache) <---- HERE ovl_set_dir_cache(d_inode(dentry), NULL); ovl_cache_free(&cache->entries); kfree(cache); } I'm not sure how they get out of sync.
Adding Miklos Szeredi who added the WARN_ON that is triggering in 4.14: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=4edb83bb1041e2f946ce36ea93f6bcd06d984bf4
*********** MASS BUG UPDATE ************** We apologize for the inconvenience. There are a large number of bugs to go through and several of them have gone stale. Due to this, we are doing a mass bug update across all of the Fedora 27 kernel bugs. Fedora 27 has now been rebased to 4.17.7-100.fc27. Please test this kernel update (or newer) and let us know if you issue has been resolved or if it is still present with the newer kernel. If you have moved on to Fedora 28, and are still experiencing this issue, please change the version to Fedora 28. If you experience different issues, please open a new bug report for those.
*********** MASS BUG UPDATE ************** This bug is being closed with INSUFFICIENT_DATA as there has not been a response in 5 weeks. If you are still experiencing this issue, please reopen and attach the relevant data from the latest kernel you are running and any data that might have been requested previously.