1797915 – [abrt] [faf] g_ascii_strtoll() for "10" failed with errno=11 (Resource temporarily unavailable)

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1797915 - [abrt] [faf] g_ascii_strtoll() for "10" failed with errno=11 (Resource temporarily unavailable)

Summary: [abrt] [faf] g_ascii_strtoll() for "10" failed with errno=11 (Resource tempor...

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Enterprise Linux 7
Classification:	Red Hat
Component:	NetworkManager
Sub Component:
Version:	7.8
Hardware:	Unspecified
OS:	Unspecified
Priority:	unspecified
Severity:	unspecified
Target Milestone:	rc
Target Release:	---
Assignee:	Thomas Haller
QA Contact:	Desktop QE
Docs Contact:
URL:	https://faf.lab.eng.brq.redhat.com/fa...
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2020-02-04 08:28 UTC by Vladimir Benes
Modified:	2020-09-29 20:30 UTC (History)
CC List:	8 users (show)
Fixed In Version:	NetworkManager-1.18.6-2.el7
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2020-09-29 20:30:25 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)
backtrace from https://faf.lab.eng.brq.redhat.com/faf/reports/15962/ (28.68 KB, text/plain) 2020-02-10 14:55 UTC, Thomas Haller	no flags	Details
dist-git for "glib2-2.56.1-6" to reproduce issue (3.14 KB, patch) 2020-05-04 20:41 UTC, Thomas Haller	no flags	Details \| Diff
View All

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Product Errata	RHSA-2020:4003	0	None	None	None	2020-09-29 20:30:52 UTC

Description Vladimir Benes 2020-02-04 08:28:28 UTC

This bug has been created based on an anonymous crash report requested by the package maintainer.

Report URL: https://faf.lab.eng.brq.redhat.com/faf/reports/bthash/174421a9a5c74c353f2b14c55e973df4785eae18/

Comment 2 Thomas Haller 2020-02-10 14:55:14 UTC

Created attachment 1662161 [details]
backtrace from https://faf.lab.eng.brq.redhat.com/faf/reports/15962/

Comment 3 Thomas Haller 2020-02-10 15:46:56 UTC

it's unclear how this assertion failure can happen.

(also, it can only happen in test-builds (--with-more-asserts). In production builds -- i.e. official RPMs in brew -- this cannot happen because the assertion is disabled).


I pushed https://gitlab.freedesktop.org/NetworkManager/NetworkManager/commit/90c5d1d99cea8a8a8edafb0c02f421979cf56b37 which might help finding the issue if it happens again.


Also, this might be very well some form of memory corruption, so with the provided information it's unclear how exactly... Let's wait for it to happen again.

Comment 4 Vladimir Benes 2020-02-10 16:14:38 UTC

It's actually happened 5 times already.

Comment 5 Beniamino Galvani 2020-02-21 16:07:27 UTC

Added new debug messages:

https://gitlab.freedesktop.org/NetworkManager/NetworkManager/commit/cf6940665dac3f5c62782acd590d5fca7e2cd76e

Comment 6 Vladimir Benes 2020-03-02 13:09:59 UTC

here again:
 01 Mar 2020

Comment 7 Vladimir Benes 2020-03-04 09:18:15 UTC

and again today: 
 04 Mar 2020

Comment 8 Vladimir Benes 2020-03-04 09:19:59 UTC

is there anything more visible from the new logs build?

Comment 9 Beniamino Galvani 2020-03-04 10:04:33 UTC

(In reply to Vladimir Benes from comment #8)
> is there anything more visible from the new logs build?

No, unfortunately the FAF report only includes a tar.gz for the first crash, which is

  NetworkManager-1.23.1-24939.9a971849b5.el7-ccpp-2020-02-03-19-29-56-28501.tar.gz 

I can't find backtraces for later ones. Do you know if it's possible to 'forget' this report, so that a new one will be created with the new backtrace and tarball?

Comment 10 Vladimir Benes 2020-03-25 20:31:26 UTC

(In reply to Beniamino Galvani from comment #9)
> (In reply to Vladimir Benes from comment #8)
> > is there anything more visible from the new logs build?
> 
> No, unfortunately the FAF report only includes a tar.gz for the first crash,
> which is
> 
>  
> NetworkManager-1.23.1-24939.9a971849b5.el7-ccpp-2020-02-03-19-29-56-28501.
> tar.gz 
> 
> I can't find backtraces for later ones. Do you know if it's possible to
> 'forget' this report, so that a new one will be created with the new
> backtrace and tarball?

we should see more here:
http://faf.lab.eng.brq.redhat.com/faf/reports/16800/

Comment 11 Vladimir Benes 2020-03-26 10:13:06 UTC

we have one more from today:
https://faf.lab.eng.brq.redhat.com/faf/reports/16810/

Comment 12 Thomas Haller 2020-03-28 12:29:54 UTC

at https://beaker.engineering.redhat.com/recipes/8060079#task108127080 :
2020-03-25, build 1.23.2-25251.fbb65de32e.el7

> [ 4622.324332] gsm-r5s2-01.wlan.rhts.eng.bos.redhat.com restraintd[1616]: (process:11071): libnm-ERROR **: 11:41:12.860: unexpected assertion failure: could parse "10.16.122.90" as 10.16.122.90, but not accepted by legacy parser: invalid token '10'



at https://beaker.engineering.redhat.com/recipes/8067937#task108212265
2020-03-27, 1.23.2-25274.54aaf240d2.el7

[ 2302.670735] gsm-r5s2-01.wlan.rhts.eng.bos.redhat.com restraintd[1618]: (process:22999): libnm-ERROR **: 11:03:00.967: unexpected assertion failure: could parse "10.16.122.90" as 10.16.122.90, but not accepted by legacy parser: invalid token '10'



at https://beaker.engineering.redhat.com/recipes/8070652#task108243276
2020-03-28, 1.23.2-25275.c84a4579b2.el7

[ 2399.713680] gsm-r5s9-01.wlan.rhts.eng.bos.redhat.com restraintd[1656]: (process:15978): libnm-ERROR **: 03:01:17.455: unexpected assertion failure: could parse "10.16.122.97" as 10.16.122.97, but not accepted by legacy parser: invalid token '10'




Still no idea.

Comment 13 Thomas Haller 2020-04-01 12:34:31 UTC

WIP for workaround here: https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/merge_requests/456

Comment 14 Thomas Haller 2020-04-01 12:38:15 UTC

via http://faf.lab.eng.brq.redhat.com/faf/reports/16845/ ,
https://beaker.engineering.redhat.com/recipes/8080594#task108346166 we get

(process:13347): libnm-ERROR **: 12:35:44.434: g_ascii_strtoll() for "10" failed with errno=11 (Resource temporarily unavailable) and v=10

Comment 15 Thomas Haller 2020-04-02 05:55:17 UTC

workaround merged to

master:  https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/commit/d342fa267d59cd317cc3cc798824c70b1158ff61
nm-1-22: https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/commit/85811e67ddcdeb47a88aca177c75436f75d5e79e

Let's see if that avoids the crash.


Then we should see whether to backport it further and/or to RHEL, and how to fix it in glib.

Comment 16 Thomas Haller 2020-05-04 20:41:56 UTC

Created attachment 1684971 [details]
dist-git for "glib2-2.56.1-6" to reproduce issue

Seems the crashes only happened on RHEL-7, and the workaround to NetworkManager worked.

To further identify where the issue comes from, let's test with a modified glib2 package (see attached dist-git patch).

@Vladimir, please run our glib2 tests with scratch-build https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=28362505, if that causes new crashes, we can see where the issue is.

Comment 20 Vladimir Benes 2020-05-13 09:40:36 UTC

not visible anymore

Comment 22 errata-xmlrpc 2020-09-29 20:30:25 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: NetworkManager security and bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:4003

Note You need to log in before you can comment on or make changes to this bug.