Bug 2049010 - Firewalld silently fails leaving system unprotected
Summary: Firewalld silently fails leaving system unprotected
Keywords:
Status: CLOSED EOL
Alias: None
Product: Fedora
Classification: Fedora
Component: firewalld
Version: 35
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: ---
Assignee: Eric Garver
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-02-01 10:34 UTC by David
Modified: 2022-12-13 16:31 UTC (History)
3 users (show)

Fixed In Version:
Clone Of:
Environment:
Last Closed: 2022-12-13 16:31:36 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)
5.19 rc4 local kernel configuration file (132.33 KB, text/plain)
2022-07-27 14:07 UTC, stan
no flags Details
A past failure log from /var/log/ (1.01 MB, text/plain)
2022-07-27 14:33 UTC, stan
no flags Details

Description David 2022-02-01 10:34:45 UTC
Description of problem:
I found out that firewalld was not running on my PC by chance, trying to set up APACHE and noticing that I was able to reach it from other devices before enabling HTTP service in firewalld.

Version-Release number of selected component (if applicable):
1.0.1

How reproducible:
The actual cause for firewalld failure, in my case, was because of a non-existing service in the config file. It is probably sufficient to add any other non-existing service to trigger the issue.

Steps to Reproduce:
1. add a non-existing service to a firewall zone configuration
2. reboot
3. check if (that) firewalld is not running

Actual results:
Firewalld silently fails, leaving the system without a firewall. As you can imagine, this can represent some real danger, depending on the use case, and anyway strongly affects the security reputation of the overall system.

Expected results:
Firewalld could and should do anything except failing in such a case. For example loading a default valid configuration, and/or notifying the user.

Additional info:

Sample journalctl output:

------------------------------------------------------
gen 16 14:58:02 localhost.localdomain firewalld[917]: ERROR: Failed to load zone file '/etc/firewalld/zones/public.xml': INVALID_SERVICE: kde-connect
gen 16 14:58:02 localhost.localdomain firewalld[917]: ERROR: 'python-nftables' failed: internal:0:0-0: Error: No such file or directory; did you mean chain ‘filter_IN_libvirt’ in table inet ‘firewalld’?
gen 16 14:58:02 localhost.localdomain firewalld[917]: ERROR: COMMAND_FAILED: 'python-nftables' failed: internal:0:0-0: Error: No such file or directory; did you mean chain ‘filter_IN_libvirt’ in table inet ‘firewalld’?

gen 17 09:49:26 localhost.localdomain firewalld[901]: ERROR: Failed to load zone file '/etc/firewalld/zones/public.xml': INVALID_SERVICE: kde-connect
gen 17 09:49:27 localhost.localdomain firewalld[901]: ERROR: 'python-nftables' failed: internal:0:0-0: Error: No such file or directory; did you mean chain ‘filter_IN_libvirt’ in table inet ‘firewalld’?
gen 17 09:49:27 localhost.localdomain firewalld[901]: ERROR: COMMAND_FAILED: 'python-nftables' failed: internal:0:0-0: Error: No such file or directory; did you mean chain ‘filter_IN_libvirt’ in table inet ‘firewalld’?

.....

feb 01 10:56:44 localhost.localdomain firewalld[920]: ERROR: Failed to load zone file '/etc/firewalld/zones/public.xml': INVALID_SERVICE: kde-connect
feb 01 10:56:44 localhost.localdomain firewalld[920]: ERROR: 'python-nftables' failed: internal:0:0-0: Error: No such file or directory; did you mean chain ‘filter_IN_libvirt’ in table inet ‘firewalld’?
feb 01 10:56:44 localhost.localdomain firewalld[920]: ERROR: 'python-nftables' failed: internal:0:0-0: Error: No such file or directory
feb 01 10:56:44 localhost.localdomain firewalld[920]: ERROR: COMMAND_FAILED: 'python-nftables' failed: internal:0:0-0: Error: No such file or directory
------------------------------------------

status of firewalld BEFORE my intervention:
-------------------------------------------
[dave997@localhost ~]$ systemctl status firewalld.service 
● firewalld.service - firewalld - dynamic firewall daemon
     Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2022-01-31 11:57:57 CET; 23h ago
       Docs: man:firewalld(1)
   Main PID: 920 (firewalld)
      Tasks: 4 (limit: 19000)
     Memory: 39.3M
        CPU: 2.179s
     CGroup: /system.slice/firewalld.service
             └─920 /usr/bin/python3 -s /usr/sbin/firewalld --nofork --nopid

feb 01 10:56:37 localhost.localdomain firewalld[920]: ERROR: UNKNOWN_INTERFACE: 'wlp3s0' is not in any zone
feb 01 10:56:40 localhost.localdomain firewalld[920]: ERROR: 'python-nftables' failed: internal:0:0-0: Error: No such file or directory
                                                      
                                                      internal:0:0-0: Error: No such file or directory
                                                      
                                                      internal:0:0-0: Error: No such file or directory
                                                      
                                                      internal:0:0-0: Error: No such file or directory
                                                      
                                                      internal:0:0-0: Error: No such file or directory
                                                      
                                                      internal:0:0-0: Error: No such file or directory
                                                      
                                                      internal:0:0-0: Error: No such file or directory

[dave997@localhost ~]$ sudo firewall-cmd --state
failed

--------------------------

AFTER deleting the non-existing service:
--------------------------
[dave997@localhost ~]$ systemctl status firewalld.service 
● firewalld.service - firewalld - dynamic firewall daemon
     Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled; vendor preset: enabled)
     Active: active (running) since Tue 2022-02-01 11:08:46 CET; 24min ago
       Docs: man:firewalld(1)
   Main PID: 21973 (firewalld)
      Tasks: 4 (limit: 19000)
     Memory: 26.0M
        CPU: 1.607s
     CGroup: /system.slice/firewalld.service
             └─21973 /usr/bin/python3 -s /usr/sbin/firewalld --nofork --nopid

feb 01 11:08:46 localhost.localdomain systemd[1]: Starting firewalld - dynamic firewall daemon...
feb 01 11:08:46 localhost.localdomain systemd[1]: Started firewalld - dynamic firewall daemon.

[dave997@localhost ~]$ sudo firewall-cmd --state
running
---------------------------



Also, I think the above mentioned "kde-connect" service simply is the previous naming of what now is "kdeconnect" service, thus indicating that this problem might have been going on for a long time.

Comment 1 Eric Garver 2022-02-01 13:15:48 UTC
> Actual results:
> Firewalld silently fails, leaving the system without a firewall. As you can
> imagine, this can represent some real danger, depending on the use case, and
> anyway strongly affects the security reputation of the overall system.

firewalld always starts because users expect firewall-cmd (and firewall-config) to work. The UIs communicate with firewalld over dbus. This only works if firewalld is running.

There is firewall-offline-cmd, but many user don't even know that exists.

> Expected results:
> Firewalld could and should do anything except failing in such a case. For
> example loading a default valid configuration

Loading the stock config on a user config failure is the plan, but it hasn't been implemented yet. Additionally firewall-cmd could always check the status of the daemon and throw a warning if it's failed.

> and/or notifying the user.

There is a log and --state returns "failed".

How else do you expect this to be communicated?

Comment 2 David 2022-02-01 16:09:09 UTC
Ok I read my comment back and I apologize if it sounded harsh, I didn't mean to.

> firewalld always starts because users expect firewall-cmd (and
> firewall-config) to work. The UIs communicate with firewalld over dbus. This
> only works if firewalld is running.
> There is firewall-offline-cmd, but many user don't even know that exists.

What I meant was that the user was left with no working firewall. In the end it happened by chance that I was using HTTP, but it could have really been something (and somebody) else ...
I understand that firewalld was running, as I used firewall-config and saw systemctl output, but nonetheless it was ineffective.

> Loading the stock config on a user config failure is the plan, but it hasn't
> been implemented yet.

This is such great news, thank you for letting me know.

> Additionally firewall-cmd could always check the
> status of the daemon and throw a warning if it's failed.
> There is a log and --state returns "failed".

Yeah I understand this but the problem of logs is they're silent, you find out as soon as you dig into them. But maybe you don't, or maybe you do and get lost among thousands of (often useless) lines

> How else do you expect this to be communicated?

I really do know too little of firewalld to suggest. Off the bat I'd say by trying to integrate with DEs notification systems maybe, or by exploiting systemd's OnFailure feature, if I remember correctly there is one.
But anyway the point is, if I (generic user) have no firewall protecting my system, I should be warned immediately because the chances I find out by myself (in time) are few.

Comment 3 stan 2022-07-25 01:49:52 UTC
I'm seeing this in rawhide on 20220724.

systemctl shows this:
firewalld.service                            loaded    active   running firewalld - dynamic firewall daemon

The journal shows this (reverse order, most recent at top):
Jul 24 18:31:23 fedora NetworkManager[8767]: <warn>  [1658712683.2261] firewalld: [f0eb441c454b1f30,change:"enp2s0"]: complete: request failed (COMMAND_FAILED: 'python-nftables' failed: 
                                             JSON blob:
                                             {"nftables": [{"metainfo": {"json_schema_version": 1}}, {"add": {"chain": {"family": "inet", "table": "firewalld", "name": "filter_IN_public"}}}, {"add": {"chain": >
Jul 24 18:31:23 fedora firewalld[9012]: ERROR: COMMAND_FAILED: 'python-nftables' failed: 
                                        JSON blob:
                                        {"nftables": [{"metainfo": {"json_schema_version": 1}}, {"add": {"table": {"family": "inet", "name": "firewalld"}}}]}
Jul 24 18:31:23 fedora firewalld[9012]: ERROR: 'python-nftables' failed: 
                                        JSON blob:
                                        {"nftables": [{"metainfo": {"json_schema_version": 1}}, {"add": {"table": {"family": "inet", "name": "firewalld"}}}]}
Jul 24 18:31:22 fedora audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=firewalld comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? te>
Jul 24 18:31:22 fedora systemd[1]: Started firewalld.service - firewalld - dynamic firewall daemon.
Jul 24 18:31:22 fedora systemd[1]: Starting firewalld.service - firewalld - dynamic firewall daemon...

Comment 4 Eric Garver 2022-07-25 13:42:25 UTC
PSA:

firewalld 1.2.0 (present in f37) added a startup failsafe. So a failure to load the user's configuration triggers a fallback to the default configuration. The idea is to keep the host protected even if an unexpected failure occurs (as this bug reports).

Also not, this is change is not suitable for backport.

Upstream PR: https://github.com/firewalld/firewalld/pull/931

--

comment 0 (david) indicates an invalid configuration which is better handled in newer firewalld.

comment 3 (stan) indicates that the firewall backend (nftables) is broken. Possibly due to a missing package, missing kernel modules, selinux denials, etc.

I don't see this bug being addressed in f35.

Comment 5 stan 2022-07-25 15:27:06 UTC
Thanks for your help.  I am running rawhide, and it is as updated as conflicts will allow, and I still have firewalld-1.1.1-1.fc37.noarch installed.  When I look at the blocked updates I see that firewalld is probably not updating because 

Problem 71: package fail2ban-firewalld-0.11.2-12.fc37.noarch requires fail2ban-server = 0.11.2-12.fc37, but none of the providers can be installed
  - package fail2ban-server-0.11.2-12.fc37.noarch requires python(abi) = 3.10, but none of the providers can be installed
  - package python3-3.10.5-2.fc37.x86_64 requires python3-libs(x86-64) = 3.10.5-2.fc37, but none of the providers can be installed
  - cannot install both python3-libs-3.11.0~b4-1.fc37.x86_64 and python3-libs-3.10.5-2.fc37.x86_64
  - package hexchat-2.16.0-5.fc37.x86_64 requires libpython3.11.so.1.0()(64bit), but none of the providers can be installed
  - cannot install the best update candidate for package hexchat-2.16.0-3.fc36.x86_64
  - cannot install the best update candidate for package fail2ban-firewalld-0.11.2-12.fc37.noarch

There are many packages not updating because of conflicts.  After the mass rebuild has ended, I will open bugzillas for those that seem important because they are blocking updates, so that might take care of this.

I have python3-nftables-1.0.4-1.fc37.x86_64 and nftables-1.0.4-1.fc37.x86_64 installed.  As near as I can tell from the package lists, there is no python-nftables.  There is an nftables.py in python3-nftables, and that is all.  I couldn't find any link to that package called python-nftables.  So, there might be a naming issue in the call in firewalld.

Can you explain a little more about the default configuration?  Does it still block inbound connections that aren't a response to an outbound connection?  That is, is there still some firewall blocking actually happening?  It might be easier to just describe what it does.

Comment 6 Eric Garver 2022-07-25 15:34:37 UTC
(In reply to stan from comment #5)
> Can you explain a little more about the default configuration?  Does it
> still block inbound connections that aren't a response to an outbound
> connection?  That is, is there still some firewall blocking actually
> happening?  It might be easier to just describe what it does.

It is exactly the configuration you get from a fresh firewalld install. That means the stock configuration shipped in /usr/lib/firewalld.

Comment 7 stan 2022-07-25 15:52:29 UTC
I interpret the information there, with my limited understanding, as saying that the only inbound connections allowed are for the services in /usr/lib/firewalld/services.  If I don't have those services active, or it isn't for one of those services, the packet will be dropped.  That works for me.  The only vulnerability relevant for me that I see is that people can hit the browser without being related to an outbound connection.  I still want the full firewall, but I don't feel as naked any more.  Thanks.

Comment 8 stan 2022-07-25 15:59:25 UTC
I might change the firewall in the router to drop some of those services that I won't ever be using as well.  Belt and suspenders.

Comment 9 Eric Garver 2022-07-25 16:40:48 UTC
(In reply to stan from comment #7)
> I interpret the information there, with my limited understanding, as saying
> that the only inbound connections allowed are for the services in
> /usr/lib/firewalld/services.  If I don't have those services active, or it
> isn't for one of those services, the packet will be dropped.

Not exactly. Those services are available in the stock configuration. But they're not _active_ (accepting traffic) until they're used in a zone or policy. Most notably the default zone will have some active services, e.g. ssh.

Comment 10 stan 2022-07-25 18:06:13 UTC
That's even better, only three active services in the public zone, which is the default.

  <service name="ssh"/>
  <service name="mdns"/>
  <service name="dhcpv6-client"/>

with ssh really the only service of concern.  I already have ssh under control (disabled).

Comment 11 Eric Garver 2022-07-25 19:49:41 UTC
(In reply to stan from comment #10)
> That's even better, only three active services in the public zone, which is
> the default.

public is the default zone upstream. But some distributions, e.g. Fedora, include their own default zones, e.g. FedoraWorkstation and FedoraServer.

Comment 12 Eric Garver 2022-07-25 20:01:56 UTC
(In reply to Eric Garver from comment #11)
> (In reply to stan from comment #10)
> > That's even better, only three active services in the public zone, which is
> > the default.
> 
> public is the default zone upstream. But some distributions, e.g. Fedora,
> include their own default zones, e.g. FedoraWorkstation and FedoraServer.

Correction: The startup failsafe will _always_ fallback to "public" as the default zone; regardless of what the distribution has set for DefaultZone.

The default zone is usually determined by DefaultZone in /etc/firewalld/firewalld.conf. Distributions change the default zone [1] but the failsafe uses defaults built into the source code [2].

[1]: https://src.fedoraproject.org/rpms/firewalld/blob/rawhide/f/firewalld.spec#_135
[2]: https://github.com/firewalld/firewalld/blob/e1aacbf2b22be1e1435e4afe3b13248e0b37c0d9/src/firewall/config/__init__.py.in#L124-L137

Comment 13 stan 2022-07-26 21:46:11 UTC
The mass rebuild finished, and I removed the packages that were blocking updates and received ~4500 updates, among them the new firewalld version, firewalld-1.2.0-2.fc37.noarch.  After rebooting to make sure that all the changes were used, firewalld still fails with the same problem, it isn't finding python-nftables.  Though the final message is different.

Jul 26 14:38:33 fedora systemd[1]: firewalld.service: Deactivated successfully.
Jul 26 14:38:33 fedora firewalld[18095]: ERROR: Raising SystemExit in run_server
Jul 26 14:38:33 fedora firewalld[18095]: ERROR: 'python-nftables' failed: 
                                         JSON blob:
                                         {"nftables": [{"metainfo": {"json_schema_version": 1}}, {"delete": {"table": {"family": "inet", "name": "firewalld"}}}]}
Jul 26 14:38:33 fedora firewalld[18095]: ERROR: Failed to load full stock configuration. This likely indicates a system level issue, e.g. the firewall backend (nftables, iptables) is broken. All hope is lost. >
Jul 26 14:38:33 fedora firewalld[18095]: ERROR: COMMAND_FAILED: 'python-nftables' failed: 
                                         JSON blob:
                                         {"nftables": [{"metainfo": {"json_schema_version": 1}}, {"add": {"table": {"family": "inet", "name": "firewalld"}}}]}
Jul 26 14:38:33 fedora firewalld[18095]: ERROR: 'python-nftables' failed: 
                                         JSON blob:
                                         {"nftables": [{"metainfo": {"json_schema_version": 1}}, {"delete": {"table": {"family": "inet", "name": "firewalld"}}}, {"add": {"table": {"family": "inet", "name": "fi>
Jul 26 14:38:33 fedora firewalld[18095]: ERROR: Failed to load user configuration. Falling back to full stock configuration.
Jul 26 14:38:33 fedora firewalld[18095]: ERROR: 'python-nftables' failed: 
                                         JSON blob:
                                         {"nftables": [{"metainfo": {"json_schema_version": 1}}, {"add": {"table": {"family": "inet", "name": "firewalld"}}}]}
Jul 26 14:38:33 fedora systemd[1]: Started firewalld.service - firewalld - dynamic firewall daemon.
Jul 26 14:38:32 fedora systemd[1]: Starting firewalld.service - firewalld - dynamic firewall daemon...

These are the contents of the python3-nftables package, python3-nftables-1.0.4-2.fc37.x86_64.

python3-nftables          /usr/lib/python3.11/site-packages/nftables
python3-nftables          /usr/lib/python3.11/site-packages/nftables-0.1-py3.11.egg-info
python3-nftables          /usr/lib/python3.11/site-packages/nftables/__init__.py
python3-nftables          /usr/lib/python3.11/site-packages/nftables/__pycache__
python3-nftables          /usr/lib/python3.11/site-packages/nftables/__pycache__/__init__.cpython-311.opt-1.pyc
python3-nftables          /usr/lib/python3.11/site-packages/nftables/__pycache__/__init__.cpython-311.pyc
python3-nftables          /usr/lib/python3.11/site-packages/nftables/__pycache__/nftables.cpython-311.opt-1.pyc
python3-nftables          /usr/lib/python3.11/site-packages/nftables/__pycache__/nftables.cpython-311.pyc
python3-nftables          /usr/lib/python3.11/site-packages/nftables/nftables.py
python3-nftables          /usr/lib/python3.11/site-packages/nftables/schema.json

Comment 14 Eric Garver 2022-07-27 12:55:38 UTC
@stan, that looks like firewalld successfully imports python3-nftables. But creating the "firewalld" tables in nftables fails. That's basically step one of applying the rule set.


Are you using a non-standard kernel? Custom build? If so, can you attach your .config (/boot/config-<kernel ver>)?

Comment 15 stan 2022-07-27 14:07:15 UTC
Created attachment 1899669 [details]
5.19 rc4 local kernel configuration file

This is from the custom kernel that is running.

Comment 16 stan 2022-07-27 14:22:13 UTC
I only thought the mass rebuild updates had occurred.  It was only the backlog.  Another 15 GB of updates from the mass rebuild happened last night.  I tried restarting firewalld with these new updates, and the same error.  In thinking about it, I'm not sure what custom settings I would have.  As far as I know I don't have any since I switched from a custom firewall to firewalld, a long time ago.  I'll check if there is some cruft hanging around in /home, since I have reinstalled fedora since I started using firewalld, so there wouldn't be any cruft left around from before that in the system.

I see these, but I don't think they are a problem:
lrwxrwxrwx. 1 0 0 system_u:object_r:etc_t:s0 22 Jun 24  2021 /etc/alternatives/iptables -> /usr/sbin/iptables-nft
lrwxrwxrwx. 1 0 0 system_u:object_r:etc_t:s0 30 Jun 24  2021 /etc/alternatives/iptables-restore -> /usr/sbin/iptables-nft-restore
lrwxrwxrwx. 1 0 0 system_u:object_r:etc_t:s0 27 Jun 24  2021 /etc/alternatives/iptables-save -> /usr/sbin/iptables-nft-save

The actual targets are from iptables-nft-1.8.8-3.fc37.x86_64, and that was installed today.

Comment 17 stan 2022-07-27 14:33:54 UTC
Created attachment 1899682 [details]
A past failure log from /var/log/

In looking around I found these failure logs from firewalld.  It seems that it started failing with the 5.18 kernel, which I have attached.

Comment 18 Eric Garver 2022-07-27 14:53:45 UTC
(In reply to stan from comment #15)
> Created attachment 1899669 [details]
> 5.19 rc4 local kernel configuration file
> 
> This is from the custom kernel that is running.

# CONFIG_NF_TABLES is not set

You need to build a kernel with support for nftables. I suggest enabling (=m) everything related to nftables.

Comment 19 stan 2022-07-27 15:58:58 UTC
Good catch!  I never even thought to look at this, because I just leave it configured the same as the previous kernel.  I know I didn't turn those off.  They must have come as default with 5.18, and I didn't change them from the old configuration.  Still, egg on face, such a simple and obvious error.  I'm in the process of building a kernel, and will confirm whether it now works, though it seems almost certain it will.

Comment 20 stan 2022-07-27 17:42:35 UTC
I compiled them into the kernel, but unfortunately I am having problems booting new kernels on my rawhide system.  Neither the custom kernels or the stock fedora kernels will boot.  I am booting a kernel compiled a few weeks ago.  I was hoping that this was the issue (a long shot), but it didn't boot.  So, I am going to consider this fixed, as it seems pretty obvious.

Thanks for your help.

PS "I just leave it configured the same as the previous kernel" makes it sound like I don't do the olcdonfig / menuconfig process.  Every time, just use the previous config as the starting point.

Comment 21 stan 2022-08-07 17:03:22 UTC
I was finally able to get a kernel with nfttables configured properly to boot, and that does fix the failure of firewalld to start successfully.  Closing this bugzilla.

Comment 22 Ben Cotton 2022-11-29 17:48:03 UTC
This message is a reminder that Fedora Linux 35 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora Linux 35 on 2022-12-13.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
'version' of '35'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, change the 'version' 
to a later Fedora Linux version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora Linux 35 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora Linux, you are encouraged to change the 'version' to a later version
prior to this bug being closed.

Comment 23 Ben Cotton 2022-12-13 16:31:36 UTC
Fedora Linux 35 entered end-of-life (EOL) status on 2022-12-13.

Fedora Linux 35 is no longer maintained, which means that it
will not receive any further security or bug fix updates. As a result we
are closing this bug.

If you can reproduce this bug against a currently maintained version of Fedora Linux
please feel free to reopen this bug against that version. Note that the version
field may be hidden. Click the "Show advanced fields" button if you do not see
the version field.

If you are unable to reopen this bug, please file a new report against an
active release.

Thank you for reporting this bug and we are sorry it could not be fixed.


Note You need to log in before you can comment on or make changes to this bug.