Created attachment 1107753 [details] sosreport of F23 system Description of problem: Stopping firewalld service via systemctl takes too long and it eventually times out Version-Release number of selected component (if applicable): F23 + latest updates as of Dec 19, 2015 How reproducible: [rsriniva@valhalla ~] $ sudo systemctl status firewalld ● firewalld.service - firewalld - dynamic firewall daemon Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled; vendor preset: enabled) Active: active (running) since Sat 2015-12-19 12:09:49 IST; 8h ago Main PID: 854 (firewalld) CGroup: /system.slice/firewalld.service └─854 /usr/bin/python3 -Es /usr/sbin/firewalld --nofork --nopid Dec 19 12:09:49 valhalla systemd[1]: Starting firewalld - dynamic firewall daemon... Dec 19 12:09:49 valhalla systemd[1]: Started firewalld - dynamic firewall daemon. [rsriniva@valhalla ~] $ sudo systemctl stop firewalld [rsriniva@valhalla ~] $ sudo systemctl status firewalld -l ● firewalld.service - firewalld - dynamic firewall daemon Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled; vendor preset: enabled) Active: failed (Result: timeout) since Sat 2015-12-19 20:47:48 IST; 24s ago Process: 854 ExecStart=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS (code=killed, signal=KILL) Main PID: 854 (code=killed, signal=KILL) CGroup: /system.slice/firewalld.service └─21753 /sbin/rmmod nf_conntrack Dec 19 12:09:49 valhalla systemd[1]: Starting firewalld - dynamic firewall daemon... Dec 19 12:09:49 valhalla systemd[1]: Started firewalld - dynamic firewall daemon. Dec 19 20:43:18 valhalla systemd[1]: Stopping firewalld - dynamic firewall daemon... Dec 19 20:44:48 valhalla systemd[1]: firewalld.service: State 'stop-sigterm' timed out. Killing. Dec 19 20:44:48 valhalla systemd[1]: firewalld.service: Main process exited, code=killed, status=9/KILL Dec 19 20:46:18 valhalla systemd[1]: firewalld.service: State 'stop-final-sigterm' timed out. Killing. Dec 19 20:47:48 valhalla systemd[1]: firewalld.service: Processes still around after final SIGKILL. Entering failed mode. Dec 19 20:47:48 valhalla systemd[1]: Stopped firewalld - dynamic firewall daemon. Dec 19 20:47:48 valhalla systemd[1]: firewalld.service: Unit entered failed state. Dec 19 20:47:48 valhalla systemd[1]: firewalld.service: Failed with result 'timeout'. Actual results: command times out after a long time - 3-4 mins Expected results: graceful halt of firewalld Additional info: [rsriniva@valhalla ~] $ rpm -qa | grep firewalld firewalld-filesystem-0.3.14.2-4.fc23.noarch firewalld-0.3.14.2-4.fc23.noarch [rsriniva@valhalla ~] $ uname -a Linux valhalla 4.2.7-300.fc23.x86_64 #1 SMP Wed Dec 9 22:28:30 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux [rsriniva@valhalla ~] $
The issue is that "rmmod nf_conntrack" hangs. This is a kernel issue. Please attach the log about the kernel oops from the system log.
Hi, Nothing in the kernel logs. Are you able to repro this on your machine? I yum removed abrt-* because it was very annoying. How else can I generate kernel oops logs? Can I turn on debug level logging for firewalld to understand what is happening?
I am not able to reproduce this on my F-23 machines so far. Are you able to restart firewalld in the running system with "systemctl restart firewalld.service"? In common there is a kernel oops if rmmod hangs on unloading a netfilter module, but it does not seem to be the case for you. Are you doing special things over the network? I have not seen an issue with rmmod nf_conntrack since releases. Do you also have the issue with using an 4.2.6 or 4.2.8 kernel?
restart of firewalld service also failed. Dec 21 21:18:25 valhalla systemd[1]: Stopping firewalld - dynamic firewall daemon... Dec 21 21:18:25 valhalla audit: NETFILTER_CFG table=mangle family=2 entries=42 Dec 21 21:18:25 valhalla audit: NETFILTER_CFG table=mangle family=2 entries=28 Dec 21 21:18:25 valhalla audit: NETFILTER_CFG table=mangle family=2 entries=6 Dec 21 21:18:25 valhalla audit: NETFILTER_CFG table=nat family=2 entries=67 Dec 21 21:18:25 valhalla audit: NETFILTER_CFG table=nat family=2 entries=37 Dec 21 21:18:25 valhalla audit: NETFILTER_CFG table=nat family=2 entries=5 Dec 21 21:18:25 valhalla audit: NETFILTER_CFG table=raw family=2 entries=9 Dec 21 21:18:25 valhalla audit: NETFILTER_CFG table=raw family=2 entries=7 Dec 21 21:18:25 valhalla audit: NETFILTER_CFG table=raw family=2 entries=3 Dec 21 21:18:25 valhalla audit: NETFILTER_CFG table=security family=2 entries=13 Dec 21 21:18:25 valhalla audit: NETFILTER_CFG table=security family=2 entries=10 Dec 21 21:18:25 valhalla audit: NETFILTER_CFG table=security family=2 entries=4 Dec 21 21:18:25 valhalla audit: NETFILTER_CFG table=filter family=2 entries=112 Dec 21 21:18:25 valhalla audit: NETFILTER_CFG table=filter family=2 entries=48 Dec 21 21:18:25 valhalla audit: NETFILTER_CFG table=filter family=2 entries=4 Dec 21 21:18:25 valhalla audit: NETFILTER_CFG table=mangle family=10 entries=40 Dec 21 21:18:25 valhalla audit: NETFILTER_CFG table=mangle family=10 entries=28 Dec 21 21:18:25 valhalla audit: NETFILTER_CFG table=mangle family=10 entries=6 Dec 21 21:18:25 valhalla audit: NETFILTER_CFG table=nat family=10 entries=52 Dec 21 21:18:26 valhalla audit: NETFILTER_CFG table=nat family=10 entries=35 Dec 21 21:18:26 valhalla audit: NETFILTER_CFG table=nat family=10 entries=5 Dec 21 21:18:26 valhalla audit: NETFILTER_CFG table=raw family=10 entries=11 Dec 21 21:18:26 valhalla audit: NETFILTER_CFG table=raw family=10 entries=7 Dec 21 21:18:26 valhalla audit: NETFILTER_CFG table=raw family=10 entries=3 Dec 21 21:18:26 valhalla audit: NETFILTER_CFG table=security family=10 entries=13 Dec 21 21:18:26 valhalla audit: NETFILTER_CFG table=security family=10 entries=10 Dec 21 21:18:26 valhalla audit: NETFILTER_CFG table=security family=10 entries=4 Dec 21 21:18:26 valhalla audit: NETFILTER_CFG table=filter family=10 entries=87 Dec 21 21:18:26 valhalla audit: NETFILTER_CFG table=filter family=10 entries=46 Dec 21 21:18:26 valhalla audit: NETFILTER_CFG table=filter family=10 entries=4 Dec 21 21:18:26 valhalla audit: NETFILTER_CFG table=nat family=7 entries=0 Dec 21 21:18:26 valhalla audit: NETFILTER_CFG table=nat family=7 entries=0 Dec 21 21:18:26 valhalla audit: NETFILTER_CFG table=nat family=7 entries=0 Dec 21 21:18:26 valhalla audit: NETFILTER_CFG table=broute family=7 entries=0 Dec 21 21:18:26 valhalla audit: NETFILTER_CFG table=broute family=7 entries=0 Dec 21 21:18:26 valhalla audit: NETFILTER_CFG table=broute family=7 entries=0 Dec 21 21:18:26 valhalla audit: NETFILTER_CFG table=filter family=7 entries=0 Dec 21 21:18:26 valhalla audit: NETFILTER_CFG table=filter family=7 entries=0 Dec 21 21:18:26 valhalla audit: NETFILTER_CFG table=filter family=7 entries=0 Dec 21 21:18:26 valhalla audit: NETFILTER_CFG table=mangle family=2 entries=6 Dec 21 21:18:26 valhalla audit: NETFILTER_CFG table=mangle family=2 entries=6 Dec 21 21:18:26 valhalla audit: NETFILTER_CFG table=mangle family=2 entries=6 Dec 21 21:18:26 valhalla audit: NETFILTER_CFG table=mangle family=2 entries=6 Dec 21 21:18:26 valhalla audit: NETFILTER_CFG table=mangle family=2 entries=6 Dec 21 21:18:26 valhalla audit: NETFILTER_CFG table=raw family=2 entries=3 Dec 21 21:18:26 valhalla audit: NETFILTER_CFG table=raw family=2 entries=3 Dec 21 21:18:26 valhalla audit: NETFILTER_CFG table=security family=2 entries=4 Dec 21 21:18:26 valhalla audit: NETFILTER_CFG table=security family=2 entries=4 Dec 21 21:18:26 valhalla audit: NETFILTER_CFG table=security family=2 entries=4 Dec 21 21:18:26 valhalla audit: NETFILTER_CFG table=filter family=2 entries=4 Dec 21 21:18:26 valhalla audit: NETFILTER_CFG table=filter family=2 entries=4 Dec 21 21:18:26 valhalla audit: NETFILTER_CFG table=filter family=2 entries=4 Dec 21 21:18:26 valhalla audit: NETFILTER_CFG table=mangle family=10 entries=6 Dec 21 21:18:26 valhalla audit: NETFILTER_CFG table=mangle family=10 entries=6 Dec 21 21:18:26 valhalla audit: NETFILTER_CFG table=mangle family=10 entries=6 Dec 21 21:18:26 valhalla audit: NETFILTER_CFG table=mangle family=10 entries=6 Dec 21 21:18:26 valhalla audit: NETFILTER_CFG table=mangle family=10 entries=6 Dec 21 21:18:26 valhalla audit: NETFILTER_CFG table=raw family=10 entries=3 Dec 21 21:18:26 valhalla audit: NETFILTER_CFG table=raw family=10 entries=3 Dec 21 21:18:26 valhalla audit: NETFILTER_CFG table=security family=10 entries=4 Dec 21 21:18:26 valhalla audit: NETFILTER_CFG table=security family=10 entries=4 Dec 21 21:18:26 valhalla audit: NETFILTER_CFG table=security family=10 entries=4 Dec 21 21:18:26 valhalla audit: NETFILTER_CFG table=filter family=10 entries=4 Dec 21 21:18:26 valhalla audit: NETFILTER_CFG table=filter family=10 entries=4 Dec 21 21:18:26 valhalla audit: NETFILTER_CFG table=filter family=10 entries=4 Dec 21 21:18:26 valhalla audit: NETFILTER_CFG table=nat family=7 entries=0 Dec 21 21:18:26 valhalla audit: NETFILTER_CFG table=nat family=7 entries=0 Dec 21 21:18:26 valhalla audit: NETFILTER_CFG table=nat family=7 entries=0 Dec 21 21:18:26 valhalla audit: NETFILTER_CFG table=broute family=7 entries=0 Dec 21 21:18:26 valhalla audit: NETFILTER_CFG table=filter family=7 entries=0 Dec 21 21:18:26 valhalla audit: NETFILTER_CFG table=filter family=7 entries=0 Dec 21 21:18:26 valhalla audit: NETFILTER_CFG table=filter family=7 entries=0 Dec 21 21:19:10 valhalla chronyd[855]: Selected source 202.71.140.36 Dec 21 21:19:56 valhalla systemd[1]: firewalld.service: State 'stop-sigterm' timed out. Killing. Dec 21 21:19:56 valhalla systemd[1]: firewalld.service: Main process exited, code=killed, status=9/KILL Dec 21 21:19:56 valhalla docker[3525]: time="2015-12-21T21:19:56.024488774+05:30" level=info msg="Firewalld running: false" Dec 21 21:19:56 valhalla libvirtd[1043]: The name org.fedoraproject.FirewallD1 was not provided by any .service files Dec 21 21:19:56 valhalla libvirtd[1043]: The name org.fedoraproject.FirewallD1 was not provided by any .service files Dec 21 21:20:49 valhalla systemd[1]: Starting dnf makecache... Dec 21 21:20:49 valhalla dnf[17426]: cachedir: /var/cache/dnf Dec 21 21:20:49 valhalla dnf[17426]: Loaded plugins: builddep, debuginfo-install, noroot, playground, generate_completion_cache, reposync, config-manager, protected_packages, Query, needs-restarting, copr, download Dec 21 21:20:49 valhalla dnf[17426]: DNF version: 1.1.4 Dec 21 21:20:49 valhalla dnf[17426]: Making cache files for all metadata files. Dec 21 21:20:49 valhalla dnf[17426]: Metadata cache refreshed recently. Dec 21 21:20:49 valhalla systemd[1]: Started dnf makecache. Dec 21 21:20:49 valhalla audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=dnf-makecache comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Dec 21 21:20:49 valhalla audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=dnf-makecache comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Dec 21 21:21:26 valhalla systemd[1]: firewalld.service: State 'stop-final-sigterm' timed out. Killing. systemctl status reported something interesting: [rsriniva@valhalla ~] $ sudo systemctl status firewalld -l ● firewalld.service - firewalld - dynamic firewall daemon Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled; vendor preset: enabled) Active: active (running) since Mon 2015-12-21 21:22:56 IST; 4s ago Main PID: 17573 (firewalld) CGroup: /system.slice/firewalld.service ├─17390 /sbin/rmmod nf_conntrack └─17573 /usr/bin/python3 -Es /usr/sbin/firewalld --nofork --nopid Dec 21 21:22:58 valhalla /firewalld[17573]: 2015-12-21 21:22:58 ERROR: COMMAND_FAILED: '/sbin/iptables -w --table filter --delete OUTPUT --out-interface virbr1 --protocol udp --destination-port 68 --jump ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?). Dec 21 21:22:58 valhalla /firewalld[17573]: 2015-12-21 21:22:58 ERROR: COMMAND_FAILED: '/sbin/iptables -w --table filter --delete INPUT --in-interface virbr1 --protocol udp --destination-port 67 --jump ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?). Dec 21 21:22:58 valhalla /firewalld[17573]: 2015-12-21 21:22:58 ERROR: COMMAND_FAILED: '/sbin/iptables -w --table filter --delete INPUT --in-interface virbr1 --protocol tcp --destination-port 67 --jump ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?). Dec 21 21:22:58 valhalla /firewalld[17573]: 2015-12-21 21:22:58 ERROR: COMMAND_FAILED: '/sbin/iptables -w --table filter --insert FORWARD --destination 192.168.100.0/24 --out-interface virbr1 --match conntrack --ctstate ESTABLISHED,RELATED --jump ACCEPT' failed: iptables: No chain/target/match by that name. Dec 21 21:22:58 valhalla /firewalld[17573]: 2015-12-21 21:22:58 ERROR: COMMAND_FAILED: '/sbin/iptables -w --table nat --delete POSTROUTING --source 192.168.100.0/24 --destination 224.0.0.0/24 --jump RETURN' failed: iptables v1.4.21: can't initialize iptables table `nat': Table does not exist (do you need to insmod?) Perhaps iptables or your kernel needs to be upgraded. Dec 21 21:22:59 valhalla /firewalld[17573]: 2015-12-21 21:22:59 ERROR: COMMAND_FAILED: '/sbin/iptables -w --table nat --delete POSTROUTING --source 192.168.100.0/24 --destination 255.255.255.255/32 --jump RETURN' failed: iptables v1.4.21: can't initialize iptables table `nat': Table does not exist (do you need to insmod?) Perhaps iptables or your kernel needs to be upgraded. Dec 21 21:22:59 valhalla /firewalld[17573]: 2015-12-21 21:22:59 ERROR: COMMAND_FAILED: '/sbin/iptables -w --table nat --delete POSTROUTING --source 192.168.100.0/24 -p tcp ! --destination 192.168.100.0/24 --jump MASQUERADE --to-ports 1024-65535' failed: iptables v1.4.21: can't initialize iptables table `nat': Table does not exist (do you need to insmod?) Perhaps iptables or your kernel needs to be upgraded. Dec 21 21:22:59 valhalla /firewalld[17573]: 2015-12-21 21:22:59 ERROR: COMMAND_FAILED: '/sbin/iptables -w --table nat --delete POSTROUTING --source 192.168.100.0/24 -p udp ! --destination 192.168.100.0/24 --jump MASQUERADE --to-ports 1024-65535' failed: iptables v1.4.21: can't initialize iptables table `nat': Table does not exist (do you need to insmod?) Perhaps iptables or your kernel needs to be upgraded. Dec 21 21:22:59 valhalla /firewalld[17573]: 2015-12-21 21:22:59 ERROR: COMMAND_FAILED: '/sbin/iptables -w --table nat --delete POSTROUTING --source 192.168.100.0/24 ! --destination 192.168.100.0/24 --jump MASQUERADE' failed: iptables v1.4.21: can't initialize iptables table `nat': Table does not exist (do you need to insmod?) Perhaps iptables or your kernel needs to be upgraded. Dec 21 21:22:59 valhalla /firewalld[17573]: 2015-12-21 21:22:59 ERROR: COMMAND_FAILED: '/sbin/iptables -w --table filter --delete FORWARD --destination 192.168.100.0/24 --out-interface virbr1 --match conntrack --ctstate ESTABLISHED,RELATED --jump ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?). [rsriniva@valhalla ~] $ I am running kvm and docker [rsriniva@valhalla ~] $ brctl show bridge name bridge id STP enabled interfaces docker0 8000.02424d1d47ff no virbr0 8000.52540093337d yes virbr0-nic virbr1 8000.525400f01c5b yes virbr1-nic [rsriniva@valhalla ~] $ ifconfig docker0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500 inet 172.17.0.1 netmask 255.255.0.0 broadcast 0.0.0.0 inet6 fe80::42:4dff:fe1d:47ff prefixlen 64 scopeid 0x20<link> ether 02:42:4d:1d:47:ff txqueuelen 0 (Ethernet) RX packets 1184 bytes 64966 (63.4 KiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 1158 bytes 88178 (86.1 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536 inet 127.0.0.1 netmask 255.0.0.0 inet6 ::1 prefixlen 128 scopeid 0x10<host> loop txqueuelen 0 (Local Loopback) RX packets 2422 bytes 212084 (207.1 KiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 2422 bytes 212084 (207.1 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 virbr0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500 inet 192.168.124.1 netmask 255.255.255.0 broadcast 192.168.124.255 ether 52:54:00:93:33:7d txqueuelen 0 (Ethernet) RX packets 0 bytes 0 (0.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 0 bytes 0 (0.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 virbr1: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500 inet 192.168.100.1 netmask 255.255.255.0 broadcast 192.168.100.255 ether 52:54:00:f0:1c:5b txqueuelen 0 (Ethernet) RX packets 0 bytes 0 (0.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 0 bytes 0 (0.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 wlp3s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1492 inet 192.168.1.232 netmask 255.255.255.0 broadcast 192.168.1.255 inet6 fe80::a65e:60ff:febb:4a47 prefixlen 64 scopeid 0x20<link> ether a4:5e:60:bb:4a:47 txqueuelen 1000 (Ethernet) RX packets 4073 bytes 1726290 (1.6 MiB) RX errors 0 dropped 154 overruns 0 frame 0 TX packets 3072 bytes 436096 (425.8 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 [rsriniva@valhalla ~] $ uname -a Linux valhalla 4.2.7-300.fc23.x86_64 #1 SMP Wed Dec 9 22:28:30 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
I rebooted into older kernel 4.2.6 and stopping firewalld worked! So it is definitely something related to 4.2.7 kernel Changed component to kernel from firewalld
BTW. The iptables related errors seen in #4 is also seen in 4.2.6 when stopping and starting - so that was really not the issue
The errors after starting firewalld are from libvirt. It removes rules for clean up (--delete) when firewalld starts. The rules are not there because firewalld just started and the rules have not been created by libvirt, yet. It is not checking if the rules exist before removing them, which results in the errors.
Just upgraded to 4.2.8 today. Still same issue. Looks like something broke after 4.2.6 kernel
Even with 4.2.8 I am not able to reproduce this issue. This seems to be related to your network usage. Have you been using docker or libvirt when this happens?
Docker daemon is not running but libvirtd is enabled to run on boot. I stopped libvirtd daemon and tried stopping firewalld with same result - [rsriniva@valhalla ~] $ sudo systemctl status firewalld -l ● firewalld.service - firewalld - dynamic firewall daemon Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled; vendor preset: enabled) Active: deactivating (stop-sigterm) since Wed 2015-12-30 09:45:05 IST; 1min 5s ago Main PID: 868 (firewalld) CGroup: /system.slice/firewalld.service ├─ 868 /usr/bin/python3 -Es /usr/sbin/firewalld --nofork --nopid └─3991 /sbin/rmmod nf_conntrack Dec 30 09:39:09 valhalla systemd[1]: Starting firewalld - dynamic firewall daemon... Dec 30 09:39:10 valhalla systemd[1]: Started firewalld - dynamic firewall daemon. Dec 30 09:45:05 valhalla systemd[1]: Stopping firewalld - dynamic firewall daemon... Any other debug flags for firewalld?
Debugging firewalld is not helping here as it seems to be an issue with the kernel module nf_conntrack.
*********** MASS BUG UPDATE ************** We apologize for the inconvenience. There is a large number of bugs to go through and several of them have gone stale. Due to this, we are doing a mass bug update across all of the Fedora 23 kernel bugs. Fedora 23 has now been rebased to 4.7.4-100.fc23. Please test this kernel update (or newer) and let us know if you issue has been resolved or if it is still present with the newer kernel. If you have moved on to Fedora 24 or 25, and are still experiencing this issue, please change the version to Fedora 24 or 25. If you experience different issues, please open a new bug report for those.
*********** MASS BUG UPDATE ************** This bug is being closed with INSUFFICIENT_DATA as there has not been a response in 4 weeks. If you are still experiencing this issue, please reopen and attach the relevant data from the latest kernel you are running and any data that might have been requested previously.
I also have the same bug, I had it in Fedora 24 and it's still present in Fedora 25. It even prevent my laptop to shutdown correctly.
I opened a new bug against Fedora 25 for this: https://bugzilla.redhat.com/show_bug.cgi?id=1397274 I couldn't figure out how to re-open this one so I just filed a new one.
I'm pretty sure bugs 1294415, 1293041, and 1397274 are all related. I'm having the same issue on my RHEL7 installations (six of them).
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days