Bug 2362648
| Summary: | The newest iptables upraade breaks docker | ||
|---|---|---|---|
| Product: | [Fedora] Fedora | Reporter: | Dominik Zogg <dominik.zogg> |
| Component: | iptables | Assignee: | Zbigniew Jędrzejewski-Szmek <zbyszek> |
| Status: | MODIFIED --- | QA Contact: | Fedora Extras Quality Assurance <extras-qa> |
| Severity: | high | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 42 | CC: | admin, comp, crow, dangnho99, davek, djpeebles, egarver, fedora, fedoraproject, franceopf, hugo.posnic, kevin, philbates35, psutter, ramercer, studenik, tad, todoleza, xzhou, zbyszek |
| Target Milestone: | --- | Keywords: | Reopened |
| Target Release: | --- | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | iptables-1.8.11-7.fc42 | Doc Type: | --- |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2025-04-29 20:40:22 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
Dominik Zogg
2025-04-28 06:10:59 UTC
I set it on high, cause if someone uses docker on Fedora as a server (not like me as a deveoper machine) they will have a bad day. What is the actual problem with iptables-nft? The version 1.8.11-4 introduced with Fedora 42 breaks Dockers custom networks. It was fixed with 1.8.11-5. See: https://github.com/docker/for-linux/issues/1525#issuecomment-2815436491 Now the version 1.8.11-6 breaks Docker again completely, because Docker can't find iptables. Who is responsible for this horror? Hi @comp I use docker provided by Fedora itself, you use the packages from docker themself? Cause 1.8.11-4 works with the onces provided by Fedora. It may break one fixing the other. This are the relavant docker pages and direct deps, i got: ``` sudo dnf remove docker-cli Paket Architektur Version Paketquelle Größe Wird entfernt: docker-cli x86_64 27.5.1-1.fc42 fedora 27.9 MiB Abhängige Pakete werden entfernt: docker-buildx x86_64 0.20.1-2.fc42 updates 65.6 MiB docker-compose x86_64 2.33.1-1.fc42 fedora 68.1 MiB moby-engine x86_64 27.5.1-1.fc42 fedora 105.2 MiB Nicht benötigte Abhängigkeiten werden entfernt: docker-compose-switch x86_64 1.0.5-2.fc42 fedora 3.4 MiB moby-engine-nano noarch 27.5.1-1.fc42 fedora 102.1 KiB moby-filesystem noarch 27.5.1-1.fc42 fedora 0.0 B tini-static x86_64 0.19.0-10.fc42 fedora 773.9 KiB ``` (In reply to Dominik Zogg from comment #3) > Hi @comp > > I use docker provided by Fedora itself, you use the packages from docker > themself? > > Cause 1.8.11-4 works with the onces provided by Fedora. It may break one > fixing the other. > > This are the relavant docker pages and direct deps, i got: > > ``` > sudo dnf remove docker-cli > Paket Architektur > Version Paketquelle > Größe > Wird entfernt: > docker-cli x86_64 > 27.5.1-1.fc42 fedora > 27.9 MiB > Abhängige Pakete werden entfernt: > docker-buildx x86_64 > 0.20.1-2.fc42 updates > 65.6 MiB > docker-compose x86_64 > 2.33.1-1.fc42 fedora > 68.1 MiB > moby-engine x86_64 > 27.5.1-1.fc42 fedora > 105.2 MiB > Nicht benötigte Abhängigkeiten werden entfernt: > docker-compose-switch x86_64 > 1.0.5-2.fc42 fedora > 3.4 MiB > moby-engine-nano noarch > 27.5.1-1.fc42 fedora > 102.1 KiB > moby-filesystem noarch > 27.5.1-1.fc42 fedora > 0.0 B > tini-static x86_64 > 0.19.0-10.fc42 fedora > 773.9 KiB > ``` Yes, I use the packages from Docker, not from Fedora. And I want to stick with the "original" packages from Docker. docker-ce-3:28.1.1-1.fc42 I just downloaded iptables-nft version 1.8.11-5 from https://koji.fedoraproject.org/koji/buildinfo?buildID=2702646 and installed it manually. This also breaks firewalld when using the iptables backend: Apr 28 06:21:46 localhost firewalld[1493]: WARNING: iptables-restore and iptables are missing, disabling IPv4 firewall. Apr 28 06:21:46 localhost firewalld[1493]: WARNING: ip6tables-restore and ip6tables are missing, disabling IPv6 firewall. Apr 28 06:21:46 localhost firewalld[1493]: WARNING: ebtables-restore and ebtables are missing, disabling bridge firewall. Apr 28 06:21:48 localhost firewalld[1493]: ERROR: Failed to load user configuration. Falling back to full stock configuration. Apr 28 06:21:48 localhost firewalld[1493]: ERROR: UNKNOWN_ERROR: No IPv4 and IPv6 firewall. Can confirm. latest 1.8.11-6 once again introduces many networking bugs. Can users here share some logs? Mainly I want to see _where_ docker/firewalld are looking for iptables. There was a recent bin/sbin unification in f42 and I suspect this ticket is fallout from that. For firewalld, you can enable debug /etc/sysconfig/firewalld. Use --debug=9. Then share the log after restarting firewalld. The latest update to 8.11-6 also breaks nordvpn. After the upgrade, all attempts to create a nordvpn connection fail. firewalld is looking in eg. /usr/sbin/iptables-restore but even though rpm -ql iptables-nft shows it should contain eg. /usr/bin/iptables-restore, neither file actually exists 1.8.11-5 does contain eg. /usr/sbin/iptables-restore and actually exists on the filesystem but it is a symlink to eg. /etc/alternatives/iptables-restore It looks like the alternatives symlinks for /usr/sbin/iptables and /usr/sbin/ip6tables were removed. I was able to get docker running again by running altnernatives --config iptables and reselecting the existing choice. That recreated the symlinks. A restart of docker didn't work instantly - I had to reboot. There's a reference to a change being made in the last iptables-legacy package that seems to affect this: https://fedora.pkgs.org/42/fedora-updates-x86_64/iptables-legacy-1.8.11-6.fc42.x86_64.rpm.html 2025-04-25 - Zbigniew Jedrzejewski-Szmek <zbyszek.pl> - 1.8.11-6 - Keep symlinks managed by alternatives under /usr/bin Upon recreating the symlinks, the are in /usr/sbin, not /usr/bin. Do we really want them in /usr/bin? Running `alternatives --config iptables` fixed the issue for me as well. Hmm. Updating iptables on a fresh install of F42 does no harm apart from 'alternatives --display iptables' showing duplicate entries. When system-upgrading from F41 to F42 though (which performs the sbin-bin-merge for me), the symlinks to /etc/alternatives disappear - there is no 'iptables' nor 'iptables-restore' or 'iptables-save' anymore. FEDORA-2025-b630544731 (iptables-1.8.11-7.fc42) has been submitted as an update to Fedora 42. https://bodhi.fedoraproject.org/updates/FEDORA-2025-b630544731 FEDORA-2025-b630544731 (iptables-1.8.11-7.fc42) has been submitted as an update to Fedora 42. https://bodhi.fedoraproject.org/updates/FEDORA-2025-b630544731 FEDORA-2025-b630544731 (iptables-1.8.11-7.fc42) has been submitted as an update to Fedora 42. https://bodhi.fedoraproject.org/updates/FEDORA-2025-b630544731 FEDORA-2025-b630544731 (iptables-1.8.11-7.fc42) has been submitted as an update to Fedora 42. https://bodhi.fedoraproject.org/updates/FEDORA-2025-b630544731 FEDORA-2025-b630544731 (iptables-1.8.11-7.fc42) has been submitted as an update to Fedora 42. https://bodhi.fedoraproject.org/updates/FEDORA-2025-b630544731 FEDORA-2025-b630544731 (iptables-1.8.11-7.fc42) has been submitted as an update to Fedora 42. https://bodhi.fedoraproject.org/updates/FEDORA-2025-b630544731 Hi, today I upgraded system Fedora 42 and docker service has started to fail. >> systemctl restart docker Job for docker.service failed because the control process exited with error code. >> journalctl -f dub 29 15:26:05 fedora dockerd[9835]: failed to start daemon: Error initializing network controller: error obtaining controller instance: failed to register "bridge" driver: failed to create NAT chain DOCKER: dub 29 15:26:05 fedora systemd[1]: COMMAND_FAILED: INVALID_IPV: 'ipv4' is not a valid backend or is unavailable Failed to start docker.service - Docker Application Container Engine. iptables-libs-1.8.11-6.fc42.x86_64 iptables-nft-1.8.11-6.fc42.x86_6 FEDORA-2025-b630544731 (iptables-1.8.11-7.fc42) has been pushed to the Fedora 42 stable repository. If problem still persists, please make note of it in this bug report. I've just updated to latest (iptables-1.8.11-7.fc42) and still cannot start docker The new build should not break an upgrade but if your upgrade was broken, I don't think it will fix it. Try running the alternatives command mentioned above. I think that will fix it and hopefully future upgrades won't break it based on the recent fixes. (In reply to dangnho99 from comment #22) > I've just updated to latest (iptables-1.8.11-7.fc42) and still cannot start > docker Strange. It's ok now. After let the laptop go to suspend to go having lunch, it works. (In reply to Dave Koberstein from comment #23) > The new build should not break an upgrade but if your upgrade was broken, I > don't think it will fix it. > > Try running the alternatives command mentioned above. I think that will fix > it and hopefully future upgrades won't break it based on the recent fixes. Yes, I expected the same. The reports from testers saying it fixed their setup hence surprised me, but who knows, maybe they rolled back before testing. (In reply to dangnho99 from comment #22) > I've just updated to latest (iptables-1.8.11-7.fc42) and still cannot start > docker As suggested, you may have to reapply your alternatives config. Either way, no need to cancel my needinfo request because of that! ;) Yeah, the existing fixes were incomplete. I submitted a new round of pull requests now: https://src.fedoraproject.org/rpms/iptables/pull-request/14 https://src.fedoraproject.org/rpms/iptables/pull-request/15 It looks like this is also affecting ufw. ufw 0.35-33 (latest version) doesn't seem to be able to start correctly with iptables-nft 1.8.11-8.fc42, however downgrading to version 1.8.11-4.fc42 solves the issue. Fix is still uncomplete in iptables-nft-1.8.11-8.fc42.x86_64 Updated today from F40 to F42. Same observation: docker daemon is not starting. Logfile: ======= Jun 21 14:54:49 nas dockerd[2436]: time="2025-06-21T14:54:49.186879702+02:00" level=warning msg="failed to find iptables" error="exec: \"iptables\": executable file not found in $PATH" Jun 21 14:54:49 nas dockerd[2436]: time="2025-06-21T14:54:49.241449401+02:00" level=info msg="stopping event stream following graceful shutdown" error="<nil>" module=libcontainerd namespace=moby Packages: ========= $ rpm -qa | grep iptables-nft iptables-nft-1.8.11-8.fc42.x86_64 $ rpm -qa | grep docker-ce docker-ce-cli-28.2.2-1.fc42.x86_64 docker-ce-rootless-extras-28.2.2-1.fc42.x86_64 docker-ce-28.2.2-1.fc42.x86_64 Workaround: =========== Running as workaround command from above and reboot fixes the issue here, too $ sudo alternatives --config iptables Reopening as per comment 28. Zbigniew, this is yours now. |