RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 2029179 - fwupd daemon fails to start
Summary: fwupd daemon fails to start
Keywords:
Status: CLOSED UPSTREAM
Alias: None
Product: Red Hat Enterprise Linux 9
Classification: Red Hat
Component: fwupd
Version: CentOS Stream
Hardware: x86_64
OS: Linux
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: Richard Hughes
QA Contact: Eirik Fuller
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-12-05 15:33 UTC by Vladimir Rusinov
Modified: 2021-12-09 14:37 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-12-09 14:37:12 UTC
Type: Bug
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker RHELPLAN-104768 0 None None None 2021-12-05 15:36:08 UTC

Description Vladimir Rusinov 2021-12-05 15:33:17 UTC 
Description of problem:

fwupd does not appear to work in CentOS 9 Stream:

```
$ sudo systemctl restart fwupd
Job for fwupd.service failed because the control process exited with error code.
See "systemctl status fwupd.service" and "journalctl -xeu fwupd.service" for details.


$ systemctl status fwupd.service
× fwupd.service - Firmware update daemon
     Loaded: loaded (/usr/lib/systemd/system/fwupd.service; static)
     Active: failed (Result: exit-code) since Sun 2021-12-05 15:27:25 GMT; 45s ago
       Docs: https://fwupd.org/
    Process: 4065783 ExecStart=/usr/libexec/fwupd/fwupd (code=exited, status=1/FAILURE)
   Main PID: 4065783 (code=exited, status=1/FAILURE)
        CPU: 198ms

Dec 05 15:27:25 apu.home.rusinov.ie systemd[1]: Starting Firmware update daemon...
Dec 05 15:27:25 apu.home.rusinov.ie fwupd[4065783]: Failed to load engine: Failed to load remotes: Unable to find default local file monitor type
Dec 05 15:27:25 apu.home.rusinov.ie systemd[1]: fwupd.service: Main process exited, code=exited, status=1/FAILURE
Dec 05 15:27:25 apu.home.rusinov.ie systemd[1]: fwupd.service: Failed with result 'exit-code'.
Dec 05 15:27:25 apu.home.rusinov.ie systemd[1]: Failed to start Firmware update daemon.

$ sudo journalctl -u fwupd
Dec 05 15:27:25 apu.home.rusinov.ie fwupd[4065783]: Failed to load engine: Failed to load remotes: Unable to find default local file monitor type
Dec 05 15:27:25 apu.home.rusinov.ie systemd[1]: fwupd.service: Main process exited, code=exited, status=1/FAILURE
Dec 05 15:27:25 apu.home.rusinov.ie systemd[1]: fwupd.service: Failed with result 'exit-code'.
Dec 05 15:27:25 apu.home.rusinov.ie systemd[1]: Failed to start Firmware update daemon.
```


Version-Release number of selected component (if applicable):

```
$ rpm -qa | grep fwupd
fwupd-1.5.9-3.el9.x86_64
fwupd-plugin-flashrom-1.5.9-3.el9.x86_64

$ cat /etc/*release
CentOS Stream release 9
NAME="CentOS Stream"
VERSION="9"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="9"
PLATFORM_ID="platform:el9"
PRETTY_NAME="CentOS Stream 9"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:9"
HOME_URL="https://centos.org/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
CentOS Stream release 9
CentOS Stream release 9
```


How reproducible:

100% on my machine but overall unknown. This install is in-place upgrade from CentOS Stream 8 to 9 (roughly following this guide: https://www.tecmint.com/upgrade-centos-7-to-centos-8/). I will try fresh install now as well.

However I don't think it is missing any configuration:

```
$ sudo find /etc -name \*.rpmnew
/etc/nsswitch.conf.rpmnew
/etc/chrony.conf.rpmnew
```


Steps to Reproduce:
1. sudo systemctl restart fwupd

Actual results:

fwupd fails with `Failed to load engine: Failed to load remotes: Unable to find default local file monitor type`

Expected results:

fwupd works

Additional info:
Comment 1 Richard Hughes 2021-12-05 16:24:16 UTC 
This sounds similar to an upstream report that we never tracked down: https://github.com/fwupd/fwupd/issues/3790

Are you missing any packages compared to a system that was installed with RHEL 9 rather than upgraded from RHEL 8 to RHEL 9? i.e. is the fwupd package missing a `Requires` we normally get for "free" with another normally-installed package? Thanks.
Comment 2 Vladimir Rusinov 2021-12-05 19:00:15 UTC 
I did a fresh install of CentOS 9 Stream (latest dvd from the centos.org in libvirt VM), and fwupd service works fine in new install. so that's good news, and package is sound for most users.

I will now try to see what's the difference between the systems. I saw that issue too, so I guess I'll especially focus on things like inotify / file limits / etc. I understand in-place upgrade is not something you'd want to support but as you said this may still be a hint of a bug.
Comment 3 Vladimir Rusinov 2021-12-05 19:12:14 UTC 
It wasn't fwupd's fault after all.

A "Too many open files" error randomly popped up, which made me look into sysctl inotify limits.

fwupd service was able to start and after doing `sudo sysctl fs.inotify.max_user_instances=512`.

The failure of fwupd after 8->9 upgrade is most likely a coincidence. The affected system is running a bunch of containerd containers as part of a small kubernetes cluster, so its workload must have changed after upgrade.

I think the only thing fwupd could have done better is a better error message ("Too many open files") - I will suggest it on the linked issue.
Comment 4 Richard Hughes 2021-12-09 14:13:24 UTC 
We added this upstream; would you be okay if I closed this with NEXT_RELEASE or UPSTREAM?
Comment 5 Vladimir Rusinov 2021-12-09 14:30:20 UTC 
Sure, I would have done it myself if I had permission to do so. Thanks for your attention!
Note You need to log in before you can comment on or make changes to this bug.