Bug 1494065
| Summary: | [NMCI][abrt] [faf] NetworkManager: unknown function(): /usr/bin/nmcli killed by 6 | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | Vladimir Benes <vbenes> | ||||
| Component: | glib2 | Assignee: | Colin Walters <walters> | ||||
| Status: | CLOSED ERRATA | QA Contact: | Desktop QE <desktop-qa-list> | ||||
| Severity: | high | Docs Contact: | |||||
| Priority: | unspecified | ||||||
| Version: | 7.4 | CC: | atragler, bgalvani, fgiudici, jkoten, klember, lmiksik, lrintel, mcepl, msugaya, nobody+bgollahe, rkhan, sukulkar, thaller, tpelka, vbenes, walters, ypu | ||||
| Target Milestone: | rc | ||||||
| Target Release: | --- | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| URL: | http://faf.lab.eng.brq.redhat.com/faf/reports/bthash/5075cbf20dbdf2a172171977eb38a9d3c3ca4e6c/ | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | glib2-2.54.2-2.el7 | Doc Type: | If docs needed, set a value | ||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2018-04-10 13:04:05 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Bug Depends On: | |||||||
| Bug Blocks: | 1470965 | ||||||
| Attachments: |
|
||||||
|
Description
Vladimir Benes
2017-09-21 12:25:32 UTC
as Beniamino pointed out there may be some interesting details here: "but the backtrace looks interesting because there are 2 parallel execution of add_interfaces() and this may be an hint to the actual problem...." We see frequent crashes of nmcli in the GLib code when the object
manager client is initialized. For example:
Thread 1:
#0 0x00007f848fed0c15 in _int_malloc (av=av@entry=0x7f8470000020, bytes=bytes@entry=128) at malloc.c:3782
#1 0x00007f848fed310c in __GI___libc_malloc (bytes=128) at malloc.c:2897
#2 0x00007f849067fb66 in g_realloc (mem=0x0, n_bytes=128) at gmem.c:159
#3 0x00007f849064e07a in g_ptr_array_maybe_expand (array=array@entry=0x7f847800b4e0, len=len@entry=1) at garray.c:1118
#4 0x00007f849064f18b in g_ptr_array_add (array=0x7f847800b4e0, data=0x7f847000aec0) at garray.c:1382
#5 0x00007f849067a198 in g_main_context_check (context=context@entry=0x7f847000ad60, max_priority=0, fds=fds@entry=0x7f847000afb0, n_fds=n_fds@entry=1) at gmain.c:3817
#6 0x00007f849067a730 in g_main_context_iterate (context=0x7f847000ad60, block=block@entry=1, dispatch=dispatch@entry=1, self=<optimized out>) at gmain.c:3924
#7 0x00007f849067aaea in g_main_loop_run (loop=0x7f847000ac60) at gmain.c:4123
#8 0x00007f8490c5c23b in initable_init (initable=0x7f8470009a50, cancellable=0x0, error=0x7f8486763a98) at gdbusproxy.c:1949
#9 0x00007f8490bf31fa in g_initable_new_valist (object_type=<optimized out>, first_property_name=first_property_name@entry=0x7f8490ccaf90 "g-connection", var_args=var_args@entry=0x7f8486763910, cancellable=cancellable@entry=0x0, error=error@entry=0x7f8486763a98) at ginitable.c:228
#10 0x00007f8490bf32e9 in g_initable_new (object_type=object_type@entry=94165753751728, cancellable=cancellable@entry=0x0, error=error@entry=0x7f8486763a98, first_property_name=first_property_name@entry=0x7f8490ccaf90 "g-connection") at ginitable.c:146
#11 0x00007f8490c69d42 in add_interfaces (manager=manager@entry=0x55a4ac4e3460 [GDBusObjectManagerClient], object_path=0x7f847c06d5e0 "/org/freedesktop/NetworkManager/IP4Config/75", ifaces_and_properties=<optimized out>, name_owner=name_owner@entry=0x7f8478019310 ":1.1680") at gdbusobjectmanagerclient.c:1495
#12 0x00007f8490c6a954 in process_get_all_result (manager=0x55a4ac4e3460 [GDBusObjectManagerClient], value=<optimized out>, name_owner=0x7f8478019310 ":1.1680") at gdbusobjectmanagerclient.c:1647
#13 0x00007f8490c6b0c3 in initable_init (initable=0x55a4ac4e3460, cancellable=0x0, error=0x7f8486763d20) at gdbusobjectmanagerclient.c:1400
#14 0x00007f8490bcb0de in async_init_thread (task=0x55a4ac4e9840 [GTask], source_object=<optimized out>, task_data=<optimized out>, cancellable=<optimized out>) at gasyncinitable.c:257
...
Thread 2:
#0 0x00007f848fe881f7 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1 0x00007f848fe898e8 in __GI_abort () at abort.c:90
#2 0x00007f848fec7f47 in __libc_message (do_abort=do_abort@entry=2, fmt=fmt@entry=0x7f848ffd4608 "*** Error in `%s': %s: 0x%s ***\n") at ../sysdeps/unix/sysv/linux/libc_fatal.c:196
#3 0x00007f848fecf619 in _int_free (ar_ptr=0x7f8478000020, ptr=<optimized out>, str=0x7f848ffd46a0 "free(): invalid next size (normal)", action=3) at malloc.c:5023
#4 0x00007f848fecf619 in _int_free (av=0x7f8478000020, p=<optimized out>, have_lock=0) at malloc.c:3845
#5 0x00007f849067fbce in g_free (mem=0x7f847801ca30) at gmem.c:189
#6 0x00007f84906689e1 in g_hash_table_resize (hash_table=hash_table@entry=0x55a4ac4cb580 = {...}) at ghash.c:619
#7 0x00007f8490668fe4 in g_hash_table_insert_node (hash_table=0x55a4ac4cb580 = {...}) at ghash.c:645
#8 0x00007f8490668fe4 in g_hash_table_insert_node (hash_table=hash_table@entry=0x55a4ac4cb580 = {...}, node_index=node_index@entry=4, key_hash=key_hash@entry=3982537675, new_key=new_key@entry=0x55a4ac4e97c0, new_value=new_value@entry=0x55a4ac4e2160, keep_new_key=keep_new_key@entry=0, reusing_key=reusing_key@entry=0) at ghash.c:978
#9 0x00007f84906690e7 in g_hash_table_insert_internal (hash_table=0x55a4ac4cb580 = {...}, key=0x55a4ac4e97c0, value=0x55a4ac4e2160, keep_new_key=0) at ghash.c:1229
#10 0x00007f8490c69f4f in add_interfaces (manager=manager@entry=0x55a4ac4e3460 [GDBusObjectManagerClient], object_path=0x55a4ac4e9640 "/org/freedesktop/NetworkManager/IP6Config/85", ifaces_and_properties=<optimized out>, name_owner=0x7f8478019310 ":1.1680") at gdbusobjectmanagerclient.c:1558
#11 0x00007f8490c6a80b in on_control_proxy_g_signal (proxy=<optimized out>, sender_name=<optimized out>, signal_name=0x55a4ac4e7bb0 "InterfacesAdded", parameters=0x7f8478009070, user_data=0x55a4ac4e3460) at gdbusobjectmanagerclient.c:1672
#12 0x00007f848ea2edcc in ffi_call_unix64 () at ../src/x86/unix64.S:76
#13 0x00007f848ea2e6f5 in ffi_call (cif=cif@entry=0x7ffd3f42aea0, fn=<optimized out>, rvalue=0x7ffd3f42ae10, avalue=avalue@entry=0x7ffd3f42ad90) at ../src/x86/ffi64.c:522
#18 0x00007f849096bddf in <emit signal ??? on instance 0x7f8478004400 [GDBusProxy]> (instance=instance@entry=0x7f8478004400, signal_id=<optimized out>, detail=detail@entry=0) at gsignal.c:3447
#14 0x00007f84909521a8 in g_cclosure_marshal_generic (closure=0x7f8478019f70, return_gvalue=0x0, n_param_values=<optimized out>, param_values=<optimized out>, invocation_hint=<optimized out>, marshal_data=0x0) at gclosure.c:1490
#15 0x00007f8490951968 in g_closure_invoke (closure=0x7f8478019f70, return_value=return_value@entry=0x0, n_param_values=4, param_values=param_values@entry=0x7ffd3f42b0a0, invocation_hint=invocation_hint@entry=0x7ffd3f42b040) at gclosure.c:804
#16 0x00007f8490963a7d in signal_emit_unlocked_R (node=node@entry=0x7f8478018290, detail=detail@entry=0, instance=instance@entry=0x7f8478004400, emission_return=emission_return@entry=0x0, instance_and_params=instance_and_params@entry=0x7ffd3f42b0a0) at gsignal.c:3635
#17 0x00007f849096baf1 in g_signal_emit_valist (instance=<optimized out>, signal_id=<optimized out>, detail=<optimized out>, var_args=var_args@entry=0x7ffd3f42b260) at gsignal.c:3391
#19 0x00007f8490c5ae7c in on_signal_received (connection=<optimized out>, sender_name=0x7f847c0721f0 ":1.1680", object_path=<optimized out>, interface_name=<optimized out>, signal_name=0x7f847c0722a0 "InterfacesAdded", parameters=0x7f8478009070, user_data=0x7f8478015d60) at gdbusproxy.c:917
#20 0x00007f8490c4a7b5 in emit_signal_instance_in_idle_cb (data=data@entry=0x7f847c074950) at gdbusconnection.c:3705
#21 0x00007f84906770d7 in g_idle_dispatch (source=0x7f847c04d0d0, callback=0x7f8490c4a740 <emit_signal_instance_in_idle_cb>, user_data=0x7f847c074950) at gmain.c:5543
...
I'm attaching the full backtrace.
It seems to me that the call to:
g_hash_table_insert (manager->priv->map_object_path_to_object_proxy, ...)
in add_interfaces() in not protected by locks and can race with other
operations on the hash table in other threads.
Reassigning the bug to glib.
Created attachment 1331993 [details]
backtrace
Fixed upstream: https://git.gnome.org/browse/glib/commit/?id=aeecd81dd13e4d8ef609149f82770ad06a8fccdb Since we're seeing the crash quite often in nmcli, can the fix be included in RHEL 7.5? Ping, any news on this? We continue to see crashes caused by this bug in our CI tests. *** Bug 1450075 has been marked as a duplicate of this bug. *** Does anyone know what incantations need to be made for this to get pm_ack+? (In reply to Colin Walters from comment #7) > Does anyone know what incantations need to be made for this to get pm_ack+? Devel phase is officially over based on schedule so bugbot do not pm_ack automagicaly. You need to ask bgollahe. I backported the patch to glib2-2.54.2-2.el7 *** Bug 1512484 has been marked as a duplicate of this bug. *** *** Bug 1513368 has been marked as a duplicate of this bug. *** *** Bug 1525197 has been marked as a duplicate of this bug. *** Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:0770 |