Bug 2226577

Summary: ModemManager failed to start when booting Fedora-KDE-Live-x86_64-Rawhide-20230725.n.1.iso
Product: [Fedora] Fedora Reporter: Matt Fagnani <matt.fagnani>
Component: ModemManagerAssignee: Tao Jin <tao-j>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: medium Docs Contact:
Priority: unspecified    
Version: rawhideCC: bgalvani, davide, dcbw, lkundrak, max, redhat-bugzilla, tao-j, thaller
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Linux   
Whiteboard:
Fixed In Version: ModemManager-1.20.6-3.fc39 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-07-31 02:12:05 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 2143446    
Attachments:
Description Flags
Journal showing ModemManager failing to start when booting none

Description Matt Fagnani 2023-07-26 02:24:03 UTC
I booted Fedora-KDE-Live-x86_64-Rawhide-20230725.n.1.iso on bare metal and in a QEMU/KVM VM in GNOME Boxes in a F38 KDE Plasma installation. ModemManager failed to start both times due to undefined symbols in its files.

Jul 26 02:05:10 ModemManager[1378]: <info>  ModemManager (version 1.20.6-2.fc39) starting in system bus...
Jul 26 02:05:11 systemd[1]: systemd-rfkill.service: Deactivated successfully.
Jul 26 02:05:11 ModemManager[1378]: <warn>  [plugin-manager] could not load shared '/usr/lib64/ModemManager/libmm-shared-xmm.so': /usr/lib64/ModemManager/libmm-shared-xmm.so: undefined symbol: mm_base_modem_get_port_gps_control
Jul 26 02:05:11 ModemManager[1378]: <warn>  [plugin-manager] could not load shared '/usr/lib64/ModemManager/libmm-shared-telit.so': /usr/lib64/ModemManager/libmm-shared-telit.so: undefined symbol: mm_base_modem_response_processor_string_ignore_at_errors
Jul 26 02:05:11 ModemManager[1378]: <warn>  [plugin-manager] could not load shared '/usr/lib64/ModemManager/libmm-shared-sierra.so': /usr/lib64/ModemManager/libmm-shared-sierra.so: undefined symbol: mm_3gpp_pdp_context_active_cmp
Jul 26 02:05:11 ModemManager[1378]: <warn>  [plugin-manager] could not load shared '/usr/lib64/ModemManager/libmm-shared-option.so': /usr/lib64/ModemManager/libmm-shared-option.so: undefined symbol: mm_base_modem_peek_port_primary
Jul 26 02:05:11 ModemManager[1378]: <warn>  [plugin-manager] could not load shared '/usr/lib64/ModemManager/libmm-shared-novatel.so': /usr/lib64/ModemManager/libmm-shared-novatel.so: undefined symbol: mm_port_serial_open
Jul 26 02:05:11 ModemManager[1378]: <warn>  [plugin-manager] could not load shared '/usr/lib64/ModemManager/libmm-shared-fibocom.so': /usr/lib64/ModemManager/libmm-shared-fibocom.so: undefined symbol: mm_broadband_modem_mbim_peek_port_mbim
Jul 26 02:05:11 ModemManager[1378]: <warn>  [plugin-manager] could not load shared '/usr/lib64/ModemManager/libmm-shared-icera.so': /usr/lib64/ModemManager/libmm-shared-icera.so: undefined symbol: mm_bearer_connect_result_unref
Jul 26 02:05:11 ModemManager[1378]: <warn>  [plugin-manager] could not load shared '/usr/lib64/ModemManager/libmm-shared-foxconn.so': /usr/lib64/ModemManager/libmm-shared-foxconn.so: undefined symbol: mm_base_modem_peek_port_primary
Jul 26 02:05:11 ModemManager[1378]: <warn>  [plugin-manager] could not load plugin '/usr/lib64/ModemManager/libmm-plugin-telit.so': /usr/lib64/ModemManager/libmm-plugin-telit.so: undefined symbol: telit_custom_init
Jul 26 02:05:11 ModemManager[1378]: <warn>  [plugin-manager] could not load plugin '/usr/lib64/ModemManager/libmm-plugin-nokia.so': /usr/lib64/ModemManager/libmm-plugin-nokia.so: undefined symbol: mm_port_probe_response_processor_is_at
Jul 26 02:05:11 ModemManager[1378]: <warn>  [plugin-manager] could not load plugin '/usr/lib64/ModemManager/libmm-plugin-x22x.so': /usr/lib64/ModemManager/libmm-plugin-x22x.so: undefined symbol: mm_base_modem_peek_port_primary
Jul 26 02:05:11 ModemManager[1378]: <warn>  [plugin-manager] could not load plugin '/usr/lib64/ModemManager/libmm-plugin-samsung.so': /usr/lib64/ModemManager/libmm-plugin-samsung.so: undefined symbol: mm_plugin_get_name
Jul 26 02:05:11 ModemManager[1378]: <warn>  [plugin-manager] could not load plugin '/usr/lib64/ModemManager/libmm-plugin-broadmobi.so': /usr/lib64/ModemManager/libmm-plugin-broadmobi.so: undefined symbol: _mm_log
Jul 26 02:05:11 ModemManager[1378]: <warn>  [plugin-manager] could not load plugin '/usr/lib64/ModemManager/libmm-plugin-thuraya.so': /usr/lib64/ModemManager/libmm-plugin-thuraya.so: undefined symbol: mm_iface_modem_messaging_get_type
Jul 26 02:05:11 ModemManager[1378]: <warn>  [plugin-manager] could not load plugin '/usr/lib64/ModemManager/libmm-plugin-ublox.so': /usr/lib64/ModemManager/libmm-plugin-ublox.so: undefined symbol: mm_bearer_connect_result_unref
Jul 26 02:05:11 ModemManager[1378]: <warn>  [plugin-manager] could not load plugin '/usr/lib64/ModemManager/libmm-plugin-iridium.so': /usr/lib64/ModemManager/libmm-plugin-iridium.so: undefined symbol: mm_bearer_connect_result_unref
Jul 26 02:05:11 ModemManager[1378]: <warn>  [plugin-manager] could not load plugin '/usr/lib64/ModemManager/libmm-plugin-quectel.so': /usr/lib64/ModemManager/libmm-plugin-quectel.so: undefined symbol: mm_base_modem_response_processor_no_result_continue
Jul 26 02:05:11 ModemManager[1378]: <warn>  [plugin-manager] could not load plugin '/usr/lib64/ModemManager/libmm-plugin-huawei.so': /usr/lib64/ModemManager/libmm-plugin-huawei.so: undefined symbol: mm_base_modem_response_processor_no_result_continue
Jul 26 02:05:11 ModemManager[1378]: <warn>  [plugin-manager] could not load plugin '/usr/lib64/ModemManager/libmm-plugin-dell.so': /usr/lib64/ModemManager/libmm-plugin-dell.so: undefined symbol: mm_common_sierra_custom_init_finish
Jul 26 02:05:11 ModemManager[1378]: <warn>  [plugin-manager] could not load plugin '/usr/lib64/ModemManager/libmm-plugin-via.so': /usr/lib64/ModemManager/libmm-plugin-via.so: undefined symbol: mm_base_modem_peek_port_primary
Jul 26 02:05:11 ModemManager[1378]: <warn>  [plugin-manager] could not load plugin '/usr/lib64/ModemManager/libmm-plugin-tplink.so': /usr/lib64/ModemManager/libmm-plugin-tplink.so: undefined symbol: _mm_log
Jul 26 02:05:11 ModemManager[1378]: <warn>  [plugin-manager] could not load plugin '/usr/lib64/ModemManager/libmm-plugin-dlink.so': /usr/lib64/ModemManager/libmm-plugin-dlink.so: undefined symbol: _mm_log
Jul 26 02:05:11 ModemManager[1378]: <warn>  [plugin-manager] could not load plugin '/usr/lib64/ModemManager/libmm-plugin-ericsson-mbm.so': /usr/lib64/ModemManager/libmm-plugin-ericsson-mbm.so: undefined symbol: mm_bearer_connect_result_unref
Jul 26 02:05:11 ModemManager[1378]: <warn>  [plugin-manager] could not load plugin '/usr/lib64/ModemManager/libmm-plugin-haier.so': /usr/lib64/ModemManager/libmm-plugin-haier.so: undefined symbol: mm_plugin_get_name
Jul 26 02:05:11 livesys-main[1292]: Adding live user [  OK  ]
Jul 26 02:05:11 ModemManager[1378]: <warn>  [plugin-manager] could not load plugin '/usr/lib64/ModemManager/libmm-plugin-option-hso.so': /usr/lib64/ModemManager/libmm-plugin-option-hso.so: undefined symbol: mm_shared_option_create_sim_finish
Jul 26 02:05:11 ModemManager[1378]: <warn>  [plugin-manager] could not load plugin '/usr/lib64/ModemManager/libmm-plugin-sierra.so': /usr/lib64/ModemManager/libmm-plugin-sierra.so: undefined symbol: _mm_log
Jul 26 02:05:11 ModemManager[1378]: <warn>  [plugin-manager] could not load plugin '/usr/lib64/ModemManager/libmm-plugin-fibocom.so': /usr/lib64/ModemManager/libmm-plugin-fibocom.so: undefined symbol: mm_bearer_connect_result_unref
Jul 26 02:05:11 systemd[1]: Started udisks2.service - Disk Manager.
Jul 26 02:05:11 udisksd[1311]: Acquired the name org.freedesktop.UDisks2 on the system message bus
Jul 26 02:05:11 ModemManager[1378]: <warn>  [plugin-manager] could not load plugin '/usr/lib64/ModemManager/libmm-plugin-longcheer.so': /usr/lib64/ModemManager/libmm-plugin-longcheer.so: undefined symbol: mm_plugin_get_type
Jul 26 02:05:11 ModemManager[1378]: <warn>  [plugin-manager] could not load plugin '/usr/lib64/ModemManager/libmm-plugin-novatel.so': /usr/lib64/ModemManager/libmm-plugin-novatel.so: undefined symbol: mm_common_novatel_custom_init
Jul 26 02:05:11 ModemManager[1378]: <warn>  [plugin-manager] could not load plugin '/usr/lib64/ModemManager/libmm-plugin-anydata.so': /usr/lib64/ModemManager/libmm-plugin-anydata.so: undefined symbol: mm_base_modem_peek_port_primary
Jul 26 02:05:11 ModemManager[1378]: <warn>  [plugin-manager] could not load plugin '/usr/lib64/ModemManager/libmm-plugin-linktop.so': /usr/lib64/ModemManager/libmm-plugin-linktop.so: undefined symbol: mm_plugin_get_type
Jul 26 02:05:11 ModemManager[1378]: <warn>  [plugin-manager] could not load plugin '/usr/lib64/ModemManager/libmm-plugin-intel.so': /usr/lib64/ModemManager/libmm-plugin-intel.so: undefined symbol: mm_shared_xmm_enable_location_gathering_finish
Jul 26 02:05:11 ModemManager[1378]: <warn>  [plugin-manager] could not load plugin '/usr/lib64/ModemManager/libmm-plugin-generic.so': /usr/lib64/ModemManager/libmm-plugin-generic.so: undefined symbol: _mm_log
Jul 26 02:05:11 ModemManager[1378]: <warn>  [plugin-manager] could not load plugin '/usr/lib64/ModemManager/libmm-plugin-simtech.so': /usr/lib64/ModemManager/libmm-plugin-simtech.so: undefined symbol: mm_port_serial_open
Jul 26 02:05:11 ModemManager[1378]: <warn>  [plugin-manager] could not load plugin '/usr/lib64/ModemManager/libmm-plugin-altair-lte.so': /usr/lib64/ModemManager/libmm-plugin-altair-lte.so: undefined symbol: mm_port_probe_response_processor_is_at
Jul 26 02:05:11 ModemManager[1378]: <warn>  [plugin-manager] could not load plugin '/usr/lib64/ModemManager/libmm-plugin-wavecom.so': /usr/lib64/ModemManager/libmm-plugin-wavecom.so: undefined symbol: mm_serial_parser_v1_destroy
Jul 26 02:05:11 ModemManager[1378]: <warn>  [plugin-manager] could not load plugin '/usr/lib64/ModemManager/libmm-plugin-nokia-icera.so': /usr/lib64/ModemManager/libmm-plugin-nokia-icera.so: undefined symbol: mm_port_probe_response_processor_is_at
Jul 26 02:05:11 ModemManager[1378]: <warn>  [plugin-manager] could not load plugin '/usr/lib64/ModemManager/libmm-plugin-gosuncn.so': /usr/lib64/ModemManager/libmm-plugin-gosuncn.so: undefined symbol: _mm_log
Jul 26 02:05:11 ModemManager[1378]: <warn>  [plugin-manager] could not load plugin '/usr/lib64/ModemManager/libmm-plugin-option.so': /usr/lib64/ModemManager/libmm-plugin-option.so: undefined symbol: mm_broadband_modem_option_new
Jul 26 02:05:11 ModemManager[1378]: <warn>  [plugin-manager] could not load plugin '/usr/lib64/ModemManager/libmm-plugin-novatel-lte.so': /usr/lib64/ModemManager/libmm-plugin-novatel-lte.so: undefined symbol: mm_3gpp_network_info_list_free
Jul 26 02:05:11 livesys-main[1405]: passwd: Note: deleting a password also unlocks the password.
Jul 26 02:05:11 ModemManager[1378]: <warn>  [plugin-manager] could not load plugin '/usr/lib64/ModemManager/libmm-plugin-mtk.so': /usr/lib64/ModemManager/libmm-plugin-mtk.so: undefined symbol: mm_base_modem_peek_port_primary
Jul 26 02:05:11 ModemManager[1378]: <warn>  [plugin-manager] could not load plugin '/usr/lib64/ModemManager/libmm-plugin-pantech.so': /usr/lib64/ModemManager/libmm-plugin-pantech.so: undefined symbol: mm_iface_modem_messaging_get_type
Jul 26 02:05:11 ModemManager[1378]: <warn>  [plugin-manager] could not load plugin '/usr/lib64/ModemManager/libmm-plugin-cinterion.so': /usr/lib64/ModemManager/libmm-plugin-cinterion.so: undefined symbol: mm_port_serial_open
Jul 26 02:05:11 ModemManager[1378]: <warn>  [plugin-manager] could not load plugin '/usr/lib64/ModemManager/libmm-plugin-foxconn.so': /usr/lib64/ModemManager/libmm-plugin-foxconn.so: undefined symbol: _mm_log
Jul 26 02:05:11 ModemManager[1378]: <warn>  [plugin-manager] could not load plugin '/usr/lib64/ModemManager/libmm-plugin-qcom-soc.so': /usr/lib64/ModemManager/libmm-plugin-qcom-soc.so: undefined symbol: mm_kernel_device_get_driver
Jul 26 02:05:11 ModemManager[1378]: <warn>  [plugin-manager] could not load plugin '/usr/lib64/ModemManager/libmm-plugin-sierra-legacy.so': /usr/lib64/ModemManager/libmm-plugin-sierra-legacy.so: undefined symbol: mm_common_sierra_custom_init
Jul 26 02:05:11 ModemManager[1378]: <warn>  [plugin-manager] could not load plugin '/usr/lib64/ModemManager/libmm-plugin-zte.so': /usr/lib64/ModemManager/libmm-plugin-zte.so: undefined symbol: mm_port_probe_response_processor_is_at
Jul 26 02:05:11 ModemManager[1378]: <warn>  [plugin-manager] could not load plugin '/usr/lib64/ModemManager/libmm-plugin-motorola.so': /usr/lib64/ModemManager/libmm-plugin-motorola.so: undefined symbol: mm_plugin_get_name
Jul 26 02:05:11 ModemManager[1378]: <warn>  could not create manager: no plugins found in plugin directory '/usr/lib64/ModemManager'
Jul 26 02:05:11 systemd[1]: Started ModemManager.service - Modem Manager.
Jul 26 02:05:11 ModemManager[1378]: <info>  ModemManager is shut down
Jul 26 02:05:11 usermod[1407]: add 'liveuser' to group 'wheel'
Jul 26 02:05:11 usermod[1407]: add 'liveuser' to shadow group 'wheel'
Jul 26 02:05:11 systemd[1]: ModemManager.service: Deactivated successfully.

ModemManager tried to restart repeatedly but failed each time in the same way. ModemManager stopped restarting after a limit was reached.

systemctl status ModemManager
× ModemManager.service - Modem Manager
     Loaded: loaded (/usr/lib/systemd/system/ModemManager.service; enabled; preset: enabled)
    Drop-In: /usr/lib/systemd/system/service.d
             └─10-timeout-abort.conf
     Active: failed (Result: start-limit-hit) since Tue 2023-07-25 22:05:44 EDT; 14min ago
   Duration: 862us
    Process: 2649 ExecStart=/usr/sbin/ModemManager (code=exited, status=0/SUCCESS)
   Main PID: 2649 (code=exited, status=0/SUCCESS)
        CPU: 48ms

Jul 25 22:05:44 localhost-live ModemManager[2649]: <warn>  could not create manager: no plugins found in plugin directory '/usr/lib64/ModemManager'
Jul 25 22:05:44 localhost-live systemd[1]: Started ModemManager.service - Modem Manager.
Jul 25 22:05:44 localhost-live ModemManager[2649]: <info>  ModemManager is shut down
Jul 25 22:05:44 localhost-live systemd[1]: ModemManager.service: Deactivated successfully.
Jul 25 22:05:44 localhost-live systemd[1]: ModemManager.service: Start request repeated too quickly.
Jul 25 22:05:44 localhost-live systemd[1]: ModemManager.service: Failed with result 'start-limit-hit'.
Jul 25 22:05:44 localhost-live systemd[1]: Failed to start ModemManager.service - Modem Manager.
Jul 25 22:05:44 localhost-live systemd[1]: ModemManager.service: Start request repeated too quickly.
Jul 25 22:05:44 localhost-live systemd[1]: ModemManager.service: Failed with result 'start-limit-hit'.
Jul 25 22:05:44 localhost-live systemd[1]: Failed to start ModemManager.service - Modem Manager.


Reproducible: Always

Steps to Reproduce:
1. boot Fedora-KDE-Live-x86_64-Rawhide-20230725.n.1.iso https://koji.fedoraproject.org/koji/buildinfo?buildID=2263602 on bare metal or in a QEMU/KVM VM in GNOME Boxes in a F38 KDE Plasma installation.
2.
3.
Actual Results:  
ModemManager failed to start when booting Fedora-KDE-Live-x86_64-Rawhide-20230725.n.1.iso

Expected Results:  
ModemManager should have started normally.

ModemManager-1.20.6-2.fc39.x86_64 was installed.

Comment 1 Matt Fagnani 2023-07-26 02:26:12 UTC
Created attachment 1980002 [details]
Journal showing ModemManager failing to start when booting

Comment 2 Allan 2023-07-30 21:59:24 UTC
This update from mass rebuild results in all our phones NOT WORKING anymore.
PLS urgently revert of fix this issue.

Output is same as already provided ^^^.

Comment 3 Fedora Blocker Bugs Application 2023-07-30 22:56:57 UTC
Proposed as a Blocker for 39-final by Fedora user mattf using the blocker tracking app because:

 ModemManager from ModemManager-1.20.6-2.fc39 failed to start during boot of Fedora-KDE-Live-x86_64-Rawhide-20230725.n.1.iso due to missing symbol errors in its libraries. I downgraded to ModemManager-1.20.6-1.fc39 with sudo dnf downgrade https://kojipkgs.fedoraproject.org//packages/ModemManager/1.20.6/1.fc39/x86_64/ModemManager-1.20.6-1.fc39.x86_64.rpm https://kojipkgs.fedoraproject.org//packages/ModemManager/1.20.6/1.fc39/x86_64/ModemManager-glib-1.20.6-1.fc39.x86_64.rpm in a QEMU/KVM VM using Fedora-KDE-Live-x86_64-Rawhide-20230725.n.1.iso. ModemManager started normally with sudo systemctl start ModemManager. The F39 Final blocker criterion "All system services present after installation with one of the release-blocking package sets must start properly, unless they require hardware which is not present. " might be violated. Allan reported that their phones don't work due to this problem at https://bugzilla.redhat.com/show_bug.cgi?id=2226577#c2

Comment 4 Tao Jin 2023-07-30 23:09:59 UTC
A quick search turns out upstream may have similar issue (unable to load symbols) and it may be related to glib.

https://gitlab.freedesktop.org/mobile-broadband/ModemManager/-/issues/758

Comment 5 Matt Fagnani 2023-07-30 23:50:36 UTC
I asked that ModemManager-1.20.6-2.fc39 be untagged from rawhide at https://bugzilla.redhat.com/show_bug.cgi?id=2226577

Comment 6 Matt Fagnani 2023-07-30 23:51:53 UTC
Sorry. I asked that ModemManager-1.20.6-2.fc39 be untagged from rawhide at https://pagure.io/releng/issue/11585

Comment 7 Fedora Update System 2023-07-31 00:44:01 UTC
FEDORA-2023-d26a306d68 has been submitted as an update to Fedora 39. https://bodhi.fedoraproject.org/updates/FEDORA-2023-d26a306d68

Comment 8 Tao Jin 2023-07-31 01:04:50 UTC
glib2 version changed several times in the past two weeks. A simple rebuild solved issue (tested in qemu with KDE live 7-30 nightly build iso). You might want to wait for a day or two for the newly built package emerge on repo and the nightly iso to test.

Comment 9 Allan 2023-07-31 01:37:46 UTC
Waoh.... It really is first time I've ever seen things happening this fast on Fedora, and I'm very happy about that.
This is/WAS a big issue for us (Fedora Mobility).
I just tested the -3 update, and that fixes it, thank you.

Comment 10 Fedora Update System 2023-07-31 02:12:05 UTC
FEDORA-2023-d26a306d68 has been pushed to the Fedora 39 stable repository.
If problem still persists, please make note of it in this bug report.