Bug 2029179

Summary: fwupd daemon fails to start
Product: Red Hat Enterprise Linux 9 Reporter: Vladimir Rusinov <vladimir.rusinov>
Component: fwupdAssignee: Richard Hughes <rhughes>
Status: CLOSED UPSTREAM QA Contact: Eirik Fuller <efuller>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: CentOS StreamCC: bstinson, jwboyer, rvr
Target Milestone: rc   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-12-09 14:37:12 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Vladimir Rusinov 2021-12-05 15:33:17 UTC
Description of problem:

fwupd does not appear to work in CentOS 9 Stream:

```
$ sudo systemctl restart fwupd
Job for fwupd.service failed because the control process exited with error code.
See "systemctl status fwupd.service" and "journalctl -xeu fwupd.service" for details.


$ systemctl status fwupd.service
× fwupd.service - Firmware update daemon
     Loaded: loaded (/usr/lib/systemd/system/fwupd.service; static)
     Active: failed (Result: exit-code) since Sun 2021-12-05 15:27:25 GMT; 45s ago
       Docs: https://fwupd.org/
    Process: 4065783 ExecStart=/usr/libexec/fwupd/fwupd (code=exited, status=1/FAILURE)
   Main PID: 4065783 (code=exited, status=1/FAILURE)
        CPU: 198ms

Dec 05 15:27:25 apu.home.rusinov.ie systemd[1]: Starting Firmware update daemon...
Dec 05 15:27:25 apu.home.rusinov.ie fwupd[4065783]: Failed to load engine: Failed to load remotes: Unable to find default local file monitor type
Dec 05 15:27:25 apu.home.rusinov.ie systemd[1]: fwupd.service: Main process exited, code=exited, status=1/FAILURE
Dec 05 15:27:25 apu.home.rusinov.ie systemd[1]: fwupd.service: Failed with result 'exit-code'.
Dec 05 15:27:25 apu.home.rusinov.ie systemd[1]: Failed to start Firmware update daemon.

$ sudo journalctl -u fwupd
Dec 05 15:27:25 apu.home.rusinov.ie fwupd[4065783]: Failed to load engine: Failed to load remotes: Unable to find default local file monitor type
Dec 05 15:27:25 apu.home.rusinov.ie systemd[1]: fwupd.service: Main process exited, code=exited, status=1/FAILURE
Dec 05 15:27:25 apu.home.rusinov.ie systemd[1]: fwupd.service: Failed with result 'exit-code'.
Dec 05 15:27:25 apu.home.rusinov.ie systemd[1]: Failed to start Firmware update daemon.
```


Version-Release number of selected component (if applicable):

```
$ rpm -qa | grep fwupd
fwupd-1.5.9-3.el9.x86_64
fwupd-plugin-flashrom-1.5.9-3.el9.x86_64

$ cat /etc/*release
CentOS Stream release 9
NAME="CentOS Stream"
VERSION="9"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="9"
PLATFORM_ID="platform:el9"
PRETTY_NAME="CentOS Stream 9"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:9"
HOME_URL="https://centos.org/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
CentOS Stream release 9
CentOS Stream release 9
```


How reproducible:

100% on my machine but overall unknown. This install is in-place upgrade from CentOS Stream 8 to 9 (roughly following this guide: https://www.tecmint.com/upgrade-centos-7-to-centos-8/). I will try fresh install now as well.

However I don't think it is missing any configuration:

```
$ sudo find /etc -name \*.rpmnew
/etc/nsswitch.conf.rpmnew
/etc/chrony.conf.rpmnew
```


Steps to Reproduce:
1. sudo systemctl restart fwupd

Actual results:

fwupd fails with `Failed to load engine: Failed to load remotes: Unable to find default local file monitor type`

Expected results:

fwupd works

Additional info:

Comment 1 Richard Hughes 2021-12-05 16:24:16 UTC
This sounds similar to an upstream report that we never tracked down: https://github.com/fwupd/fwupd/issues/3790

Are you missing any packages compared to a system that was installed with RHEL 9 rather than upgraded from RHEL 8 to RHEL 9? i.e. is the fwupd package missing a `Requires` we normally get for "free" with another normally-installed package? Thanks.

Comment 2 Vladimir Rusinov 2021-12-05 19:00:15 UTC
I did a fresh install of CentOS 9 Stream (latest dvd from the centos.org in libvirt VM), and fwupd service works fine in new install. so that's good news, and package is sound for most users.

I will now try to see what's the difference between the systems. I saw that issue too, so I guess I'll especially focus on things like inotify / file limits / etc. I understand in-place upgrade is not something you'd want to support but as you said this may still be a hint of a bug.

Comment 3 Vladimir Rusinov 2021-12-05 19:12:14 UTC
It wasn't fwupd's fault after all.

A "Too many open files" error randomly popped up, which made me look into sysctl inotify limits.

fwupd service was able to start and after doing `sudo sysctl fs.inotify.max_user_instances=512`.

The failure of fwupd after 8->9 upgrade is most likely a coincidence. The affected system is running a bunch of containerd containers as part of a small kubernetes cluster, so its workload must have changed after upgrade.

I think the only thing fwupd could have done better is a better error message ("Too many open files") - I will suggest it on the linked issue.

Comment 4 Richard Hughes 2021-12-09 14:13:24 UTC
We added this upstream; would you be okay if I closed this with NEXT_RELEASE or UPSTREAM?

Comment 5 Vladimir Rusinov 2021-12-09 14:30:20 UTC
Sure, I would have done it myself if I had permission to do so. Thanks for your attention!