Bug 2047788

Description of problem:
https://tools.dqe.lab.eng.bos.redhat.com/vbenes/nm_ci_stats/stats.html#project:beaker-NetworkManager-main-veth-rhel9-upstream;search:dhcp-timeout_infinity_timeout

the latest report is here:
https://desktopqe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/beaker-NetworkManager-main-veth-rhel9-upstream/884/artifact/artifacts/FAIL_report_NetworkManager-ci_Test0391_dhcp-timeout_infinity.html

there seems to be a crash in the after scenario:
                Found module NetworkManager with build-id: ddb90e25f1b4eb918788f948629b8d124e67b129
                Stack trace of thread 209998:
                #0  0x00007f23be5257fc __pthread_kill_implementation (libc.so.6 + 0x8f7fc)
                #1  0x00007f23be4d8676 __GI_raise (libc.so.6 + 0x42676)
                #2  0x00007f23be4c27d3 __GI_abort (libc.so.6 + 0x2c7d3)
                #3  0x00007f23be6e5d4c g_assertion_message (libglib-2.0.so.0 + 0x1ed4c)
                #4  0x00007f23be74698f g_assertion_message_expr (libglib-2.0.so.0 + 0x7f98f)
                #5  0x00005629ac937e46 _l3_acd_data_state_change (NetworkManager + 0x1fce46)
                #6  0x00005629ac939f4d _l3_acd_data_timeout_cb (NetworkManager + 0x1fef4d)
                #7  0x00007f23be71c5a1 g_timeout_dispatch (libglib-2.0.so.0 + 0x555a1)
                #8  0x00007f23be71bd4f g_main_context_dispatch (libglib-2.0.so.0 + 0x54d4f)
                #9  0x00007f23be7705c8 g_main_context_iterate.constprop.0 (libglib-2.0.so.0 + 0xa95c8)
                #10 0x00007f23be71b463 g_main_loop_run (libglib-2.0.so.0 + 0x54463)
                #11 0x00005629ac78cccd main (NetworkManager + 0x51ccd)
                #12 0x00007f23be4c3560 __libc_start_call_main (libc.so.6 + 0x2d560)
                #13 0x00007f23be4c360c __libc_start_main_impl (libc.so.6 + 0x2d60c)
                #14 0x00005629ac78d27e _start (NetworkManager + 0x5227e)
                
                Stack trace of thread 209999:
                #0  0x00007f23be59b83f __GI___poll (libc.so.6 + 0x10583f)
                #1  0x00007f23be77055c g_main_context_iterate.constprop.0 (libglib-2.0.so.0 + 0xa955c)
                #2  0x00007f23be7195d3 g_main_context_iteration (libglib-2.0.so.0 + 0x525d3)
                #3  0x00007f23be719621 glib_worker_main (libglib-2.0.so.0 + 0x52621)
                #4  0x00007f23be746d72 g_thread_proxy (libglib-2.0.so.0 + 0x7fd72)
                #5  0x00007f23be523aaf start_thread (libc.so.6 + 0x8daaf)
                #6  0x00007f23be5a8700 __clone3 (libc.so.6 + 0x112700)
                
                Stack trace of thread 210000:
                #0  0x00007f23be59b83f __GI___poll (libc.so.6 + 0x10583f)
                #1  0x00007f23be77055c g_main_context_iterate.constprop.0 (libglib-2.0.so.0 + 0xa955c)
                #2  0x00007f23be71b463 g_main_loop_run (libglib-2.0.so.0 + 0x54463)
                #3  0x00007f23be9735ca gdbus_shared_thread_func (libgio-2.0.so.0 + 0x1115ca)
                #4  0x00007f23be746d72 g_thread_proxy (libglib-2.0.so.0 + 0x7fd72)
                #5  0x00007f23be523aaf start_thread (libc.so.6 + 0x8daaf)
                #6  0x00007f23be5a8700 __clone3 (libc.so.6 + 0x112700)
                
                Stack trace of thread 210001:
                #0  0x00007f23be5a0f7d syscall (libc.so.6 + 0x10af7d)
                #1  0x00007f23be76dc1c g_cond_wait_until (libglib-2.0.so.0 + 0xa6c1c)
                #2  0x00007f23be6ec071 g_async_queue_pop_intern_unlocked (libglib-2.0.so.0 + 0x25071)
                #3  0x00007f23be74a02a g_thread_pool_thread_proxy.lto_priv.0 (libglib-2.0.so.0 + 0x8302a)
                #4  0x00007f23be746d72 g_thread_proxy (libglib-2.0.so.0 + 0x7fd72)
                #5  0x00007f23be523aaf start_thread (libc.so.6 + 0x8daaf)
                #6  0x00007f23be5a8700 __clone3 (libc.so.6 + 0x112700)

GNU gdb (GDB) Red Hat Enterprise Linux 10.2-9.el9
Copyright (C) 2021 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/sbin/NetworkManager...
Reading symbols from /usr/lib/debug/usr/sbin/NetworkManager-1.35.6-29614.copr.7895a9d7e2.fc34.x86_64.debug...
[New LWP 209998]
[New LWP 209999]
[New LWP 210000]
[New LWP 210001]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/sbin/NetworkManager --no-daemon'.
Program terminated with signal SIGABRT, Aborted.
#0  __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
44	      return INTERNAL_SYSCALL_ERROR_P (ret) ? INTERNAL_SYSCALL_ERRNO (ret) : 0;
[Current thread is 1 (Thread 0x7f23bd4124c0 (LWP 209998))]
(gdb) #0  __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
#1  0x00007f23be525863 in __pthread_kill_internal (signo=6, threadid=<optimized out>) at pthread_kill.c:78
#2  0x00007f23be4d8676 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#3  0x00007f23be4c27d3 in __GI_abort () at abort.c:79
#4  0x00007f23be6e5d4c in g_assertion_message (domain=<optimized out>, file=<optimized out>, line=<optimized out>, func=<optimized out>, message=<optimized out>) at ../glib/gtestutils.c:3065
#5  0x00007f23be74698f in g_assertion_message_expr (domain=0x5629aca70359 "nm", file=0x5629aca62aab "src/core/nm-l3cfg.c", line=2395, func=0x5629acb26b30 <__func__.72.lto_priv.4> "_l3_acd_data_state_change", expr=<optimized out>) at ../glib/gtestutils.c:3091
#6  0x00005629ac937e46 in _l3_acd_data_state_change (self=0x5629add69790, acd_data=0x5629add8d520, state_change_mode=ACD_STATE_CHANGE_MODE_TIMEOUT, sender_addr=0x0, p_now_msec=0x7ffded506460) at src/core/nm-l3cfg.c:2395
#7  0x00005629ac939f4d in _l3_acd_data_timeout_cb (user_data=user_data@entry=0x5629add8d520) at src/core/nm-l3cfg.c:1933
#8  0x00007f23be71c5a1 in g_timeout_dispatch (source=0x5629addd7a80, callback=0x5629ac939ee0 <_l3_acd_data_timeout_cb>, user_data=0x5629add8d520) at ../glib/gmain.c:4889
#9  0x00007f23be71bd4f in g_main_dispatch (context=0x5629adc6da00) at ../glib/gmain.c:3337
#10 g_main_context_dispatch (context=0x5629adc6da00) at ../glib/gmain.c:4055
#11 0x00007f23be7705c8 in g_main_context_iterate.constprop.0 (context=0x5629adc6da00, block=block@entry=1, dispatch=dispatch@entry=1, self=<optimized out>) at ../glib/gmain.c:4131
#12 0x00007f23be71b463 in g_main_loop_run (loop=0x5629adc4f470) at ../glib/gmain.c:4329
#13 0x00005629ac78cccd in main (argc=<optimized out>, argv=<optimized out>) at src/core/main.c:509
(gdb) quit


Version-Release number of selected component (if applicable):
1.35.6-29614.copr.7895a9d7e2.fc34

How reproducible:
not sure, the test is quite long, we've seen the crash just twice so far

Steps to Reproduce:
1. test=dhcp-timeout_infinity; a=0; while ./test_run.sh $test; do :;((a++)); echo "ATTEMPT $a"; if [ $a -eq 100 ]; then break; fi ; done

Actual results:
no crash

Expected results:


Additional info: