Description of problem: we have a crash in NM during openvpn_persist test: PID: 63411 (NetworkManager) UID: 0 (root) GID: 0 (root) Signal: 5 (TRAP) Timestamp: Fri 2023-06-23 08:21:36 EDT (1s ago) Command Line: /usr/sbin/NetworkManager --no-daemon Executable: /usr/sbin/NetworkManager Control Group: /system.slice/NetworkManager.service Unit: NetworkManager.service Slice: system.slice Boot ID: fcbd265819aa4365b2e4c811ab791978 Machine ID: 79eecde7ed3643e9be4e7cdb431f412f Hostname: gsm-r5s9-01.wlan.rhts.eng.bos.redhat.com Storage: /var/lib/systemd/coredump/core.NetworkManager.0.fcbd265819aa4365b2e4c811ab791978.63411.1687522896000000.zst (present) Size on Disk: 937.9K Message: Process 63411 (NetworkManager) of user 0 dumped core. Stack trace of thread 63411: #0 0x00007f767b4ab5a7 g_logv (libglib-2.0.so.0 + 0x5a5a7) #1 0x00007f767b4ab863 g_log (libglib-2.0.so.0 + 0x5a863) #2 0x000055c4c1b70de1 _dispatcher_call (NetworkManager + 0x2aede1) #3 0x000055c4c1be56e7 _set_vpn_state.constprop.0 (NetworkManager + 0x3236e7) #4 0x000055c4c1bfb540 nm_manager_deactivate_connection (NetworkManager + 0x339540) #5 0x000055c4c1bfbaf6 connection_removed (NetworkManager + 0x339af6) #6 0x00007f767ba48c7f g_closure_invoke (libgobject-2.0.so.0 + 0x13c7f) #7 0x00007f767ba64f96 signal_emit_unlocked_R (libgobject-2.0.so.0 + 0x2ff96) #8 0x00007f767ba6685a g_signal_emit_valist (libgobject-2.0.so.0 + 0x3185a) #9 0x00007f767ba66a73 g_signal_emit (libgobject-2.0.so.0 + 0x31a73) #10 0x000055c4c1b78a62 _connection_changed_delete (NetworkManager + 0x2b6a62) #11 0x000055c4c1b79458 _connection_changed_process_all_dirty (NetworkManager + 0x2b7458) #12 0x000055c4c1b8752c nm_settings_connection_delete (NetworkManager + 0x2c552c) #13 0x000055c4c1b87afd delete_auth_cb (NetworkManager + 0x2c5afd) #14 0x000055c4c1b63528 pk_auth_cb (NetworkManager + 0x2a1528) #15 0x000055c4c1a39308 _call_on_idle (NetworkManager + 0x177308) #16 0x00007f767b4a213b g_idle_dispatch (libglib-2.0.so.0 + 0x5113b) #17 0x00007f767b4a5e2f g_main_context_dispatch (libglib-2.0.so.0 + 0x54e2f) #18 0x00007f767b4fa508 g_main_context_iterate.constprop.0 (libglib-2.0.so.0 + 0xa9508) #19 0x00007f767b4a5483 g_main_loop_run (libglib-2.0.so.0 + 0x54483) #20 0x000055c4c1916b90 main (NetworkManager + 0x54b90) #21 0x00007f767b23feb0 __libc_start_call_main (libc.so.6 + 0x3feb0) #22 0x00007f767b23ff60 __libc_start_main_impl (libc.so.6 + 0x3ff60) #23 0x000055c4c1917095 _start (NetworkManager + 0x55095) Stack trace of thread 63412: #0 0x00007f767b34296f __GI___poll (libc.so.6 + 0x14296f) #1 0x00007f767b4fa49c g_main_context_iterate.constprop.0 (libglib-2.0.so.0 + 0xa949c) #2 0x00007f767b4a35f3 g_main_context_iteration (libglib-2.0.so.0 + 0x525f3) #3 0x00007f767b4a3641 glib_worker_main (libglib-2.0.so.0 + 0x52641) #4 0x00007f767b4d4582 g_thread_proxy (libglib-2.0.so.0 + 0x83582) #5 0x00007f767b29f802 start_thread (libc.so.6 + 0x9f802) #6 0x00007f767b23f450 __clone3 (libc.so.6 + 0x3f450) Stack trace of thread 63413: #0 0x00007f767b34296f __GI___poll (libc.so.6 + 0x14296f) #1 0x00007f767b4fa49c g_main_context_iterate.constprop.0 (libglib-2.0.so.0 + 0xa949c) #2 0x00007f767b4a5483 g_main_loop_run (libglib-2.0.so.0 + 0x54483) #3 0x00007f767b69be1a gdbus_shared_thread_func (libgio-2.0.so.0 + 0x110e1a) #4 0x00007f767b4d4582 g_thread_proxy (libglib-2.0.so.0 + 0x83582) #5 0x00007f767b29f802 start_thread (libc.so.6 + 0x9f802) #6 0x00007f767b23f450 __clone3 (libc.so.6 + 0x3f450) ELF object binary architecture: AMD x86-64 GNU gdb (GDB) Red Hat Enterprise Linux 10.2-10.el9 Copyright (C) 2021 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: <https://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /usr/sbin/NetworkManager... Reading symbols from /usr/lib/debug/usr/sbin/NetworkManager-1.43.10-32423.copr.8c5aec7a1b.el9.x86_64.debug... [New LWP 63411] [New LWP 63412] [New LWP 63413] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/usr/sbin/NetworkManager --no-daemon'. Program terminated with signal SIGTRAP, Trace/breakpoint trap. #0 g_logv (log_domain=0x55c4c1c5cd3c "nm", log_level=G_LOG_LEVEL_CRITICAL, format=<optimized out>, args=<optimized out>) at ../glib/gmessages.c:1413 1413 g_private_set (&g_log_depth, GUINT_TO_POINTER (depth)); [Current thread is 1 (Thread 0x7f767a18a500 (LWP 63411))] (gdb) #0 g_logv (log_domain=0x55c4c1c5cd3c "nm", log_level=G_LOG_LEVEL_CRITICAL, format=<optimized out>, args=<optimized out>) at ../glib/gmessages.c:1413 #1 0x00007f767b4ab863 in g_log (log_domain=<optimized out>, log_level=<optimized out>, format=<optimized out>) at ../glib/gmessages.c:1451 #2 0x000055c4c1b70de1 in _dispatcher_call (action=NM_DISPATCHER_ACTION_VPN_PRE_DOWN, blocking=0, device=0x0, settings_connection=0x55c4c304a130, applied_connection=0x55c4c3171bc0, activation_type_external=0, connectivity_state=NM_CONNECTIVITY_UNKNOWN, vpn_iface=0x55c4c2fbb12c "tun1", l3cd=0x55c4c30a6660, callback=0x55c4c1beb1c0 <dispatcher_pre_down_done>, user_data=0x55c4c3138900, out_call_id=0x55c4c3138998) at src/core/nm-dispatcher.c:543 #3 0x000055c4c1be56e7 in nm_dispatcher_call_vpn (out_call_id=0x55c4c3138998, user_data=0x55c4c3138900, callback=0x55c4c1beb1c0 <dispatcher_pre_down_done>, l3cd=0x55c4c30a6660, vpn_iface=0x55c4c2fbb12c "tun1", parent_device=0x0, applied_connection=0x55c4c3171bc0, settings_connection=<optimized out>, action=NM_DISPATCHER_ACTION_VPN_PRE_DOWN) at src/core/nm-dispatcher.c:846 #4 _set_vpn_state (self=0x55c4c3138900, vpn_state=<optimized out>, reason=<optimized out>, quitting=0) at src/core/vpn/nm-vpn-connection.c:1073 #5 0x000055c4c1bfb540 in nm_vpn_connection_deactivate (quitting=0, reason=NM_ACTIVE_CONNECTION_STATE_REASON_CONNECTION_REMOVED, self=0x55c4c3138900) at src/core/vpn/nm-vpn-connection.c:2255 #6 nm_manager_deactivate_connection (manager=0x55c4c2fca020, active=<optimized out>, reason=<optimized out>, error=0x7ffea9b92928) at src/core/nm-manager.c:6986 #7 0x000055c4c1bfbaf6 in nm_manager_deactivate_ac (connection=0x55c4c304a130, self=0x55c4c2fca020) at src/core/nm-manager.c:6216 #8 connection_removed (settings=<optimized out>, connection=0x55c4c304a130, user_data=<optimized out>) at src/core/nm-policy.c:2676 #12 0x00007f767ba66a73 in <emit signal ??? on instance ???> (instance=<optimized out>, signal_id=<optimized out>, detail=<optimized out>) at ../gobject/gsignal.c:3553 #9 0x00007f767ba48c7f in g_closure_invoke (closure=0x55c4c2fdd3d0, return_value=0x0, n_param_values=2, param_values=0x7ffea9b92b30, invocation_hint=0x7ffea9b92ab0) at ../gobject/gclosure.c:810 #10 0x00007f767ba64f96 in signal_emit_unlocked_R (node=node@entry=0x55c4c2fa5500, detail=detail@entry=0, instance=instance@entry=0x55c4c2f9b150, emission_return=emission_return@entry=0x0, instance_and_params=instance_and_params@entry=0x7ffea9b92b30) at ../gobject/gsignal.c:3741 #11 0x00007f767ba6685a in g_signal_emit_valist (instance=<optimized out>, signal_id=<optimized out>, detail=<optimized out>, var_args=var_args@entry=0x7ffea9b92d00) at ../gobject/gsignal.c:3497 #13 0x000055c4c1b78a62 in _emit_connection_removed (sett_conn=0x55c4c304a130, self=0x55c4c2f9b150) at src/core/settings/nm-settings.c:482 #14 _connection_changed_delete (self=0x55c4c2f9b150, storage=<optimized out>, sett_conn=0x55c4c304a130, allow_add_to_no_auto_default=<optimized out>) at src/core/settings/nm-settings.c:1252 #15 0x000055c4c1b79458 in _connection_changed_process_one (update_reason=(NM_SETTINGS_CONNECTION_UPDATE_REASON_REAPPLY_PARTIAL | NM_SETTINGS_CONNECTION_UPDATE_REASON_RESET_AGENT_SECRETS | NM_SETTINGS_CONNECTION_UPDATE_REASON_CLEAR_DEFAULT_WIRED | NM_SETTINGS_CONNECTION_UPDATE_REASON_BLOCK_AUTOCONNECT | unknown: 0x5400), override_sett_flags=0, sett_mask=NM_SETTINGS_CONNECTION_INT_FLAGS_NONE, sett_flags=3271802392, allow_add_to_no_auto_default=0, sett_conn_entry=0x55c4c2f9b150, self=<optimized out>) at src/core/settings/nm-settings.c:1310 #16 _connection_changed_process_all_dirty (self=<optimized out>, allow_add_to_no_auto_default=<optimized out>, sett_flags=<optimized out>, sett_mask=<optimized out>, override_sett_flags=<optimized out>, update_reason=<optimized out>) at src/core/settings/nm-settings.c:1351 #17 0x000055c4c1b8752c in nm_settings_delete_connection (allow_add_to_no_auto_default=-1447481296, sett_conn=<optimized out>, self=0x55c4c2f9b150) at src/core/settings/nm-settings.c:2478 #18 nm_settings_connection_delete (self=<optimized out>, allow_add_to_no_auto_default=-1447481296) at src/core/settings/nm-settings-connection.c:660 #19 0x000055c4c1b87afd in delete_auth_cb (self=0x55c4c304a130, context=0x55c4c30a8d00, subject=0x55c4c2ff9720, error=0x0, data=<optimized out>) at src/core/settings/nm-settings-connection.c:1859 #20 0x000055c4c1b63528 in pk_auth_cb (auth_manager=<optimized out>, auth_call_id=<optimized out>, is_authorized=1, is_challenge=<optimized out>, auth_error=<optimized out>, user_data=0x55c4c3164600) at src/core/settings/nm-settings-connection.c:1285 #21 0x000055c4c1a39308 in _call_id_invoke_callback (error=0x0, is_challenge=0, is_authorized=1, call_id=0x55c4c312a740) at src/core/nm-auth-manager.c:180 #22 _call_on_idle (user_data=user_data@entry=0x55c4c312a740) at src/core/nm-auth-manager.c:284 #23 0x00007f767b4a213b in g_idle_dispatch (source=0x55c4c317a610, callback=0x55c4c1a391e0 <_call_on_idle>, user_data=0x55c4c312a740) at ../glib/gmain.c:5848 #24 0x00007f767b4a5e2f in g_main_dispatch (context=0x55c4c2f8b690) at ../glib/gmain.c:3337 #25 g_main_context_dispatch (context=0x55c4c2f8b690) at ../glib/gmain.c:4055 #26 0x00007f767b4fa508 in g_main_context_iterate.constprop.0 (context=0x55c4c2f8b690, block=block@entry=1, dispatch=dispatch@entry=1, self=<optimized out>) at ../glib/gmain.c:4131 #27 0x00007f767b4a5483 in g_main_loop_run (loop=0x55c4c2f89950) at ../glib/gmain.c:4329 #28 0x000055c4c1916b90 in main (argc=<optimized out>, argv=<optimized out>) at src/core/main.c:519 Version-Release number of selected component (if applicable): NM-1.43.10 RHEL-9.2 How reproducible: always Steps to Reproduce: 1. run ./test_run.sh openvpn_persist Actual results: crash Expected results: no crash Additional info:
disabling the test for rhel9+ https://gitlab.freedesktop.org/NetworkManager/NetworkManager-ci/-/commit/8cf68cb49fc76372bab63845c0172ef018ad8712
Fixed on NetworkManager-ci repository directly.