Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1457769

Summary: [NMCI][abrt] [faf] NetworkManager: __strcmp_power8(): /usr/bin/nmcli killed by 11
Product: Red Hat Enterprise Linux 7 Reporter: Vladimir Benes <vbenes>
Component: NetworkManagerAssignee: sushil kulkarni <sukulkar>
Status: CLOSED DUPLICATE QA Contact: Desktop QE <desktop-qa-list>
Severity: medium Docs Contact:
Priority: medium    
Version: 7.4CC: atragler, bgalvani, fgiudici, lrintel, rkhan, sukulkar, thaller
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
URL: http://faf.lab.eng.brq.redhat.com/faf/reports/bthash/1808d9ff9f213c16bd09003ab1b8d1eb1832840c/
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-06-14 16:31:17 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Vladimir Benes 2017-06-01 09:27:21 UTC
This bug has been created based on an anonymous crash report requested by the package maintainer.

Report URL: http://faf.lab.eng.brq.redhat.com/faf/reports/bthash/1808d9ff9f213c16bd09003ab1b8d1eb1832840c/

Comment 1 Beniamino Galvani 2017-06-14 16:31:17 UTC

*** This bug has been marked as a duplicate of bug 1450075 ***

Comment 2 Thomas Haller 2017-06-15 08:52:38 UTC
I think there is a race.


We initialize GDBus Object Manager asyncronously. Note that this means, running sync on another thread:

static void
async_initable_iface_init (GAsyncInitableIface *async_initable_iface)
{
  /* for now, just use default: run GInitable code in thread */
}



Then, thread 2 is in process of g_initable_init(), while thread 1 receives InterfacesRemoved signal.

GDBus code is certainly broken here, because on thread 2 the initialization is like:

   - get-all-objects-sync
   - for all-objects:
        add_interface()
            take mutex & add interface

while on thread 1 it races with add_interface.


You never know whether a signal on thread 1 happened before get-all-objects-sync (and should be ignored), or after (and should be considered).

I think patch https://bugzilla.gnome.org/show_bug.cgi?id=783795 is wrong. You *must* process all signals during process_get_all_result(). But *if* you handle any (asynchrnous) signals, you *must* ignore whether the object is present in the requested synchronous get-all.

Meaning:

  if you get an InterfaceRemoved signal, you must ignore the same signal if it is present in the synchronous get-all. 





(gdb) thread 2
[Switching to thread 2 (Thread 0x3fff7f7ef010 (LWP 29564))]
#0  .syscall () at ../sysdeps/unix/sysv/linux/powerpc/syscall.S:29
29              PSEUDO_RET
(gdb) bt
#0  0x00003fff8716b8b4 in .syscall () at ../sysdeps/unix/sysv/linux/powerpc/syscall.S:29
#1  0x00003fff872f6054 in g_mutex_lock_slowpath (mutex=<optimized out>) at gthread-posix.c:1313
#2  0x00003fff87658ca0 in add_interfaces (manager=manager@entry=0x1002b1234c0 [GDBusObjectManagerClient], object_path=0x3fff7400ac20 "/org/freedesktop/NetworkManager/Settings/6", ifaces_and_properties=0x3fff7401aea0, name_owner=name_owner@entry=0x3fff70041170 ":1.10600") at gdbusobjectmanagerclient.c:1440
#3  0x00003fff87659e54 in process_get_all_result (manager=0x1002b1234c0 [GDBusObjectManagerClient], value=<optimized out>, name_owner=0x3fff70041170 ":1.10600")
    at gdbusobjectmanagerclient.c:1647
#4  0x00003fff8765a7e4 in initable_init (initable=0x1002b1234c0, cancellable=0x0, error=0x3fff7f7ee3d0) at gdbusobjectmanagerclient.c:1400
#5  0x00003fff8759c328 in g_initable_init (initable=0x1002b1234c0, cancellable=0x0, error=0x3fff7f7ee3d0) at ginitable.c:112
#6  0x00003fff87558774 in async_init_thread (task=0x1002b12f040 [GTask], source_object=<optimized out>, task_data=<optimized out>, cancellable=<optimized out>) at gasyncinitable.c:257
#7  0x00003fff875e1294 in g_task_thread_pool_thread (thread_data=0x1002b12f040, pool_data=<optimized out>) at gtask.c:1304
#8  0x00003fff87373e90 in g_thread_pool_thread_proxy (data=<optimized out>) at gthreadpool.c:307
#9  0x00003fff87372ba4 in g_thread_proxy (data=0x1002b1110a0) at gthread.c:784
#10 0x00003fff8789c95c in start_thread (arg=0x3fff7f7ef010) at pthread_create.c:310
#11 0x00003fff87173bbc in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:109
(gdb) thread 1
[Switching to thread 1 (Thread 0x3fff86a151b0 (LWP 29562))]
#0  .__strcmp_power8 () at ../sysdeps/powerpc/powerpc64/power8/strcmp.S:49
49              ld      r8,0(r3)
(gdb) bt
#0  0x00003fff8710095c in .__strcmp_power8 () at ../sysdeps/powerpc/powerpc64/power8/strcmp.S:49
#1  0x00003fff87322f00 in g_str_equal (v1=<optimized out>, v2=<optimized out>) at ghash.c:1846
#2  0x00003fff87321c30 in g_hash_table_lookup (hash_return=<synthetic pointer>, key=0x1002b12ef40, hash_table=0x1002b0f7180 = {...}) at ghash.c:396
#3  0x00003fff87321c30 in g_hash_table_lookup (hash_table=0x1002b0f7180 = {...}, key=0x1002b12ef40) at ghash.c:1147
#4  0x00003fff87659b40 in on_control_proxy_g_signal (interface_names=0x1002b1003d0, object_path=0x1002b12ef40 "/org/freedesktop/NetworkManager/ActiveConnection/25", manager=0x1002b1234c0 [GDBusObjectManagerClient]) at gdbusobjectmanagerclient.c:1580
#5  0x00003fff87659b40 in on_control_proxy_g_signal (proxy=<optimized out>, sender_name=<optimized out>, signal_name=<optimized out>, parameters=0x3fff74092f60, user_data=0x1002b1234c0)
    at gdbusobjectmanagerclient.c:1682
#6  0x00003fff86c580dc in .ffi_call_LINUX64 () at ../src/powerpc/linux64.S:116
#7  0x00003fff86c57808 in ffi_call (cif=<optimized out>, fn=<optimized out>, rvalue=0x3fffc13fd650, avalue=<optimized out>) at ../src/powerpc/ffi.c:985
#12 0x00003fff87491e18 in <emit signal ??? on instance 0x3fff7000cd10 [GDBusProxy]> (instance=<optimized out>, signal_id=<optimized out>, detail=<optimized out>) at gsignal.c:3447
    #8  0x00003fff8746e5cc in g_cclosure_marshal_generic (closure=0x3fff70040df0, return_gvalue=0x0, n_param_values=<optimized out>, param_values=<optimized out>, invocation_hint=<optimized out>, marshal_data=0x0) at gclosure.c:1490
    #9  0x00003fff8746d8c8 in g_closure_invoke (closure=0x3fff70040df0, return_value=0x0, n_param_values=<optimized out>, param_values=0x3fffc13fda40, invocation_hint=0x3fffc13fd920)
    at gclosure.c:804
    #10 0x00003fff874886b8 in signal_emit_unlocked_R (node=node@entry=0x3fff7003c720, detail=detail@entry=0, instance=instance@entry=0x3fff7000cd10, emission_return=emission_return@entry=0x0, instance_and_params=instance_and_params@entry=0x3fffc13fda40) at gsignal.c:3635
    #11 0x00003fff87491b80 in g_signal_emit_valist (instance=0x3fff7000cd10, signal_id=<optimized out>, detail=<optimized out>, var_args=<optimized out>) at gsignal.c:3391
#13 0x00003fff87641ef4 in on_signal_received (connection=<optimized out>, sender_name=0x3fff740933f0 ":1.10600", object_path=<optimized out>, interface_name=<optimized out>, signal_name=0x3fff7408f250 "InterfacesRemoved", parameters=0x3fff74092f60, user_data=<optimized out>) at gdbusproxy.c:917
#14 0x00003fff87629948 in emit_signal_instance_in_idle_cb (data=0x3fff7408eca0) at gdbusconnection.c:3705
#15 0x00003fff87336694 in g_idle_dispatch (source=0x3fff7408f140, callback=<optimized out>, user_data=<optimized out>) at gmain.c:5543
#16 0x00003fff8733bc28 in g_main_context_dispatch (context=0x1002b110e90) at gmain.c:3201
#17 0x00003fff8733bc28 in g_main_context_dispatch (context=0x1002b110e90) at gmain.c:3854
#18 0x00003fff8733c0b8 in g_main_context_iterate (context=0x1002b110e90, block=block@entry=1, dispatch=dispatch@entry=1, self=<optimized out>) at gmain.c:3927
#19 0x00003fff8733c5d0 in g_main_loop_run (loop=0x1002b1052c0) at gmain.c:4123
#20 0x0000000095985a24 in main (argc=<optimized out>, argv=<optimized out>) at clients/cli/nmcli.c:614



What I still don't understand, is how the crash could happen... :(


I think, for now we should workaround this, by initializing the object manager synchrnously.