1670111 – restraintd segfaults on ppc64le and aarch64

Bug 1670111 - restraintd segfaults on ppc64le and aarch64

Summary: restraintd segfaults on ppc64le and aarch64

Keywords:
Status:	CLOSED NEXTRELEASE
Alias:	None
Product:	Restraint
Classification:	Retired
Component:	general
Sub Component:
Version:	0.1.37
Hardware:	All
OS:	Linux
Priority:	high
Severity:	high
Target Milestone:	0.1.38
Assignee:	Bill Peck
QA Contact:
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2019-01-28 15:36 UTC by Artem Savkov
Modified:	2019-01-29 13:46 UTC (History)
CC List:	11 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2019-01-29 12:50:08 UTC
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Beaker Project Gerrit	6401	0	None	None	None	2019-01-28 20:07:29 UTC

Description Artem Savkov 2019-01-28 15:36:29 UTC

Description of problem:
restraintd segfaults during recipe fetching on ppc64le


Version-Release number of selected component (if applicable):
restraint-0.1.37-1.el8bkr.src.rpm

How reproducible:
100%

Steps to Reproduce:
1. run a job on a ppc64le machine

Actual results:
* Fetching recipe: http://<lab-controller-fqdn>//recipes/6438477/ 
[   13.542899] restraintd[5219]: *** Error in `/usr/bin/restraintd': free(): invalid pointer: 0x00003fffadfa0a10 *** 
[   13.549919] restraintd[5219]: ======= Backtrace: ========= 
[   13.551298] restraintd[5219]: /lib64/libc.so.6(cfree+0x4a0)[0x3fffae879cd0] 
[   13.552268] restraintd[5219]: /usr/bin/restraintd[0x1034b2e0] 
[   13.553188] restraintd[5219]: /usr/bin/restraintd[0x10017d50] 
[   13.554021] restraintd[5219]: /usr/bin/restraintd[0x101b215c] 
[   13.554872] restraintd[5219]: /usr/bin/restraintd[0x101dadc0] 
[   13.555986] restraintd[5219]: /usr/bin/restraintd[0x101dae30] 
[   13.556982] restraintd[5219]: /usr/bin/restraintd[0x1033e2e0] 
[   13.557828] restraintd[5219]: /usr/bin/restraintd[0x103438a0] 
[   13.558722] restraintd[5219]: /usr/bin/restraintd[0x10343d38] 
[   13.559621] restraintd[5219]: /usr/bin/restraintd[0x103442b0] 
[   13.560742] restraintd[5219]: /usr/bin/restraintd[0x10009d1c] 
[   13.561554] restraintd[5219]: /lib64/libc.so.6(+0x25100)[0x3fffae805100] 
[   13.562343] restraintd[5219]: /lib64/libc.so.6(__libc_start_main+0xc4)[0x3fffae8052f4] 
[   13.563145] restraintd[5219]: ======= Memory map: ======== 
[   13.563934] restraintd[5219]: 10000000-10830000 r-xp 00000000 fd:00 34502011                           /usr/bin/restraintd 
[   13.565074] restraintd[5219]: 10830000-10840000 r--p 00830000 fd:00 34502011                           /usr/bin/restraintd 
[   13.566238] restraintd[5219]: 10840000-10860000 rw-p 00840000 fd:00 34502011                           /usr/bin/restraintd 
[   13.567216] restraintd[5219]: 1003efc0000-1003f060000 rw-p 00000000 00:00 0                            [heap] 
[   13.568113] restraintd[5219]: 3fffa0000000-3fffa0030000 rw-p 00000000 00:00 0 
[   13.569035] restraintd[5219]: 3fffa0030000-3fffa4000000 ---p 00000000 00:00 0 
[   13.569784] restraintd[5219]: 3fffa8000000-3fffa8030000 rw-p 00000000 00:00 0 
[   13.570693] restraintd[5219]: 3fffa8030000-3fffac000000 ---p 00000000 00:00 0 
[   13.571493] restraintd[5219]: 3fffad710000-3fffad720000 r-xp 00000000 fd:00 67204907                   /usr/lib64/libnss_dns-2.17.so 
[   13.572479] restraintd[5219]: 3fffad720000-3fffad730000 r--p 00000000 fd:00 67204907                   /usr/lib64/libnss_dns-2.17.so 
[   13.573853] restraintd[5219]: 3fffad730000-3fffad740000 rw-p 00000000 00:00 0 
[   13.575054] restraintd[5219]: 3fffad740000-3fffad750000 r-xp 00000000 fd:00 67204909                   /usr/lib64/libnss_files-2.17.so 
[   13.576453] restraintd[5219]: 3fffad750000-3fffad760000 r--p 00000000 fd:00 67204909                   /usr/lib64/libnss_files-2.17.so 
[   13.577852] restraintd[5219]: 3fffad760000-3fffad780000 rw-p 00000000 00:00 0 
[   13.579181] restraintd[5219]: 3fffad780000-3fffad790000 ---p 00000000 00:00 0 
[   13.580578] restraintd[5219]: 3fffad790000-3fffadfc0000 rw-p 00000000 00:00 0 
[   13.581605] restraintd[5219]: 3fffadfc0000-3fffadfd0000 ---p 00000000 00:00 0 
[   13.582714] restraintd[5219]: 3fffadfd0000-3fffae7e0000 rw-p 00000000 00:00 0 
[   13.583635] restraintd[5219]: 3fffae7e0000-3fffae9b0000 r-xp 00000000 fd:00 67204891                   /usr/lib64/libc-2.17.so 
[   13.584671] restraintd[5219]: 3fffae9b0000-3fffae9c0000 r--p 001c0000 fd:00 67204891                   /usr/lib64/libc-2.17.so 
[   13.585898] restraintd[5219]: 3fffae9c0000-3fffae9d0000 rw-p 001d0000 fd:00 67204891                   /usr/lib64/libc-2.17.so 
[   13.586807] restraintd[5219]: 3fffae9d0000-3fffae9f0000 r-xp 00000000 fd:00 67204917                   /usr/lib64/libpthread-2.17.so 
[   13.589764] restraintd[5219]: 3fffae9f0000-3fffaea00000 r--p 00010000 fd:00 67204917                   /usr/lib64/libpthread-2.17.so 
[   13.590673] restraintd[5219]: 3fffaea00000-3fffaea10000 rw-p 00020000 fd:00 67204917                   /usr/lib64/libpthread-2.17.so 
[   13.591398] restraintd[5219]: 3fffaea10000-3fffaea30000 r-xp 00000000 fd:00 67157980                   /usr/lib64/libgcc_s-4.8.5-20150702.so.1 
[   13.592117] restraintd[5219]: 3fffaea30000-3fffaea40000 r--p 00010000 fd:00 67157980                   /usr/lib64/libgcc_s-4.8.5-20150702.so.1 
[   13.592831] restraintd[5219]: 3fffaea40000-3fffaea50000 rw-p 00020000 fd:00 67157980                   /usr/lib64/libgcc_s-4.8.5-20150702.so.1 
[   13.593556] restraintd[5219]: 3fffaea50000-3fffaea60000 r-xp 00000000 fd:00 67204925                   /usr/lib64/libutil-2.17.so 
[   13.594283] restraintd[5219]: 3fffaea60000-3fffaea70000 r--p 00000000 fd:00 67204925                   /usr/lib64/libutil-2.17.so 
[   13.595115] restraintd[5219]: 3fffaea70000-3fffaea80000 rw-p 00000000 00:00 0 
[   13.595821] restraintd[5219]: 3fffaea80000-3fffaea90000 r-xp 00000000 fd:00 67204897                   /usr/lib64/libdl-2.17.so 
[   13.596528] restraintd[5219]: 3fffaea90000-3fffaeaa0000 r--p 00000000 fd:00 67204897                   /usr/lib64/libdl-2.17.so 
[   13.597226] restraintd[5219]: 3fffaeaa0000-3fffaeab0000 rw-p 00000000 00:00 0 
[   13.597809] restraintd[5219]: 3fffaeab0000-3fffaead0000 r-xp 00000000 fd:00 67204919                   /usr/lib64/libresolv-2.17.so 
[   13.598522] restraintd[5219]: 3fffaead0000-3fffaeae0000 r--p 00010000 fd:00 67204919                   /usr/lib64/libresolv-2.17.so 
[   13.599327] restraintd[5219]: 3fffaeae0000-3fffaeaf0000 rw-p 00020000 fd:00 67204919                   /usr/lib64/libresolv-2.17.so 
[   13.600118] restraintd[5219]: 3fffaeaf0000-3fffaeb00000 r-xp 00000000 fd:00 67204921                   /usr/lib64/librt-2.17.so 
[   13.600930] restraintd[5219]: 3fffaeb00000-3fffaeb10000 r--p 00000000 fd:00 67204921                   /usr/lib64/librt-2.17.so 
[   13.601666] restraintd[5219]: 3fffaeb10000-3fffaeb20000 rw-p 00010000 fd:00 67204921                   /usr/lib64/librt-2.17.so 
[   13.602421] restraintd[5219]: 3fffaeb20000-3fffaebf0000 r-xp 00000000 fd:00 67204899                   /usr/lib64/libm-2.17.so 
[   13.603186] restraintd[5219]: 3fffaebf0000-3fffaec00000 r--p 000c0000 fd:00 67204899                   /usr/lib64/libm-2.17.so 
[   13.603957] restraintd[5219]: 3fffaec00000-3fffaec10000 rw-p 000d0000 fd:00 67204899                   /usr/lib64/libm-2.17.so 
[   13.604757] restraintd[5219]: 3fffaec10000-3fffaec20000 r--s 00000000 fd:00 100709979                  /usr/lib64/gconv/gconv-modules.cache 
[   13.606724] restraintd[5219]: 3fffaec20000-3fffaec40000 r-xp 00000000 00:00 0                          [vdso] 
[   13.607428] restraintd[5219]: 3fffaec40000-3fffaec70000 r-xp 00000000 fd:00 67204884                   /usr/lib64/ld-2.17.so 
[   13.608333] restraintd[5219]: 3fffaec70000-3fffaec80000 r--p 00020000 fd:00 67204884                   /usr/lib64/ld-2.17.so 
[   13.609067] restraintd[5219]: 3fffaec80000-3fffaec90000 rw-p 00030000 fd:00 67204884                   /usr/lib64/ld-2.17.so 
[   13.609817] restraintd[5219]: 3fffe75b0000-3fffe75e0000 rw-p 00000000 00:00 0                          [stack] 

Expected results:


Additional info:

(gdb) bt
#0  0x00007ffff7b42c6c in raise () from /lib64/libc.so.6
#1  0x00007ffff7b24520 in abort () from /lib64/libc.so.6
#2  0x00007ffff7b9007c in __libc_message () from /lib64/libc.so.6
#3  0x00007ffff7b99d48 in malloc_printerr () from /lib64/libc.so.6
#4  0x00007ffff7b9bd54 in _int_free () from /lib64/libc.so.6
#5  0x0000000010369a90 in g_free (mem=<optimized out>) at gmem.c:194
#6  0x0000000010016a5c in restraint_xml_read_callback (source=<optimized out>, result=<optimized out>, user_data=0x7ffff72d0a10)
    at xml.c:92
#7  0x00000000101c57fc in async_ready_callback_wrapper (source_object=0x1092f0e0, res=0x10964280, user_data=0x7ffff72d0a10)
    at ginputstream.c:532
#8  0x00000000101eedb0 in g_task_return_now (task=task@entry=0x10964280) at gtask.c:1148
#9  0x00000000101eee20 in complete_in_idle_cb (task=0x10964280) at gtask.c:1162
#10 0x000000001035c720 in g_idle_dispatch (source=<optimized out>, callback=<optimized out>, user_data=<optimized out>)
    at gmain.c:5535
#11 0x0000000010361a94 in g_main_dispatch (context=0x10941d00) at gmain.c:3177
#12 g_main_context_dispatch (context=context@entry=0x10941d00) at gmain.c:3830
#13 0x0000000010361f88 in g_main_context_iterate (context=0x10941d00, block=block@entry=1, dispatch=dispatch@entry=1, 
    self=<optimized out>) at gmain.c:3903
#14 0x000000001036253c in g_main_loop_run (loop=0x10962fa0) at gmain.c:4099
#15 0x0000000010008900 in main (argc=<optimized out>, argv=<optimized out>) at server.c:658

The bug does not reproduce under valgrind but it spews a couple of seemingly unrelated invalid-reads which all lead to soup_uri_new():

Waiting for client!
==31520== Invalid read of size 4
==31520==    at 0x1016E990: soup_uri_new_with_base (soup-uri.c:420)
==31520==    by 0x1016F1F3: soup_uri_new (soup-uri.c:492)
==31520==    by 0x10165597: soup_session_request (soup-session.c:4481)
==31520==    by 0x10016F47: restraint_xml_parse_from_url (xml.c:144)
==31520==    by 0x1000B5CF: recipe_handler (recipe.c:544)
==31520==    by 0x1035C71F: g_idle_dispatch (gmain.c:5535)
==31520==    by 0x10361A93: g_main_dispatch (gmain.c:3177)
==31520==    by 0x10361A93: g_main_context_dispatch (gmain.c:3830)
==31520==    by 0x10361F87: g_main_context_iterate.isra.26 (gmain.c:3903)
==31520==    by 0x1036253B: g_main_loop_run (gmain.c:4099)
==31520==    by 0x100088EB: main (server.c:658)
==31520==  Address 0x45b2061 is 17 bytes inside a block of size 19 alloc'd
==31520==    at 0x408424C: malloc (vg_replace_malloc.c:309)
==31520==    by 0x103698BF: g_malloc (gmem.c:99)
==31520==    by 0x10388853: g_strndup (gstrfuncs.c:425)
==31520==    by 0x1016C95F: uri_normalized_copy (soup-uri.c:780)
==31520==    by 0x1016E1AF: soup_uri_new_with_base (soup-uri.c:359)
==31520==    by 0x1016F1F3: soup_uri_new (soup-uri.c:492)
==31520==    by 0x10165597: soup_session_request (soup-session.c:4481)
==31520==    by 0x10016F47: restraint_xml_parse_from_url (xml.c:144)
==31520==    by 0x1000B5CF: recipe_handler (recipe.c:544)
==31520==    by 0x1035C71F: g_idle_dispatch (gmain.c:5535)
==31520==    by 0x10361A93: g_main_dispatch (gmain.c:3177)
==31520==    by 0x10361A93: g_main_context_dispatch (gmain.c:3830)
==31520==    by 0x10361F87: g_main_context_iterate.isra.26 (gmain.c:3903)
==31520== 
* Fetching recipe: http://<lab-controller-fqdn>//recipes/6438474/
==31520== Invalid read of size 4
==31520==    at 0x1016E990: soup_uri_new_with_base (soup-uri.c:420)
==31520==    by 0x1016F1F3: soup_uri_new (soup-uri.c:492)
==31520==    by 0x1000B67B: recipe_handler (recipe.c:551)
==31520==    by 0x1035C71F: g_idle_dispatch (gmain.c:5535)
==31520==    by 0x10361A93: g_main_dispatch (gmain.c:3177)
==31520==    by 0x10361A93: g_main_context_dispatch (gmain.c:3830)
==31520==    by 0x10361F87: g_main_context_iterate.isra.26 (gmain.c:3903)
==31520==    by 0x1036253B: g_main_loop_run (gmain.c:4099)
==31520==    by 0x100088EB: main (server.c:658)
==31520==  Address 0x470a981 is 17 bytes inside a block of size 19 alloc'd
==31520==    at 0x408424C: malloc (vg_replace_malloc.c:309)
==31520==    by 0x103698BF: g_malloc (gmem.c:99)
==31520==    by 0x10388853: g_strndup (gstrfuncs.c:425)
==31520==    by 0x1016C95F: uri_normalized_copy (soup-uri.c:780)
==31520==    by 0x1016E1AF: soup_uri_new_with_base (soup-uri.c:359)
==31520==    by 0x1016F1F3: soup_uri_new (soup-uri.c:492)
==31520==    by 0x1000B67B: recipe_handler (recipe.c:551)
==31520==    by 0x1035C71F: g_idle_dispatch (gmain.c:5535)
==31520==    by 0x10361A93: g_main_dispatch (gmain.c:3177)
==31520==    by 0x10361A93: g_main_context_dispatch (gmain.c:3830)
==31520==    by 0x10361F87: g_main_context_iterate.isra.26 (gmain.c:3903)
==31520==    by 0x1036253B: g_main_loop_run (gmain.c:4099)
==31520==    by 0x100088EB: main (server.c:658)
==31520==

Comment 5 Jeff Bastian 2019-01-28 19:27:33 UTC

It's failing on aarch64 too:

~]# rpm -q restraint
restraint-0.1.37-1.el8bkr.aarch64

~]# restraintd
Server listening on port 8081 for ipV6
Server listening on port 8081 for ipV4
Waiting for client!
* Fetching recipe: http://REDACTED/
free(): invalid pointer
Aborted (core dumped)

Comment 6 Jeff Bastian 2019-01-28 19:33:59 UTC

The backtrace on aarch64 is the same as ppc64le:

(gdb) bt
#0  0x0000ffffbe432d0c in raise () from /lib64/libc.so.6
#1  0x0000ffffbe4208e8 in abort () from /lib64/libc.so.6
#2  0x0000ffffbe46d554 in __libc_message () from /lib64/libc.so.6
#3  0x0000ffffbe473adc in malloc_printerr () from /lib64/libc.so.6
#4  0x0000ffffbe475378 in _int_free () from /lib64/libc.so.6
#5  0x000000000053a0c0 in async_ready_callback_wrapper (
    source_object=0xadf0e0, res=0xb16a80, user_data=0xffffbdbd0a10)
    at ginputstream.c:532
#6  0x0000000000555b80 in g_task_return_now (task=0xb16a80) at gtask.c:1148
#7  0x0000000000555bbc in complete_in_idle_cb (task=0xb16a80) at gtask.c:1162
#8  0x000000000064fa24 in g_main_dispatch (context=0xaf1570) at gmain.c:3177
#9  g_main_context_dispatch (context=context@entry=0xaf1570) at gmain.c:3830
#10 0x000000000064fdf0 in g_main_context_iterate (context=0xaf1570, 
    block=block@entry=1, dispatch=dispatch@entry=1, self=<optimized out>)
    at gmain.c:3903
#11 0x0000000000650188 in g_main_loop_run (loop=0xb21e60) at gmain.c:4099
#12 0x0000000000407620 in main (argc=<optimized out>, argv=<optimized out>)
    at server.c:658

Comment 7 Bill Peck 2019-01-28 20:07:29 UTC

I have a test repo for rhel8 here:

http://file.bos.redhat.com/~bpeck/restraint/el8--test/

I'll build for rhel7 next.

Comment 8 Bill Peck 2019-01-28 21:19:19 UTC

Here is the test repo for rhel7

http://file.bos.redhat.com/~bpeck/restraint/el7bkr--test/

      <repos>
        <repo name="myharness" url="http://file.bos.redhat.com/~bpeck/restraint/el7bkr--test/"/>
      </repos>

Comment 9 Jeff Bastian 2019-01-28 22:04:09 UTC

Thanks for the quick fix, Bill!  I tested 3 systems and they all worked fine (still waiting on a free POWER9 system to test RHEL8):

restraint-0.1.37-1.git.5.6992e65.el7bkr.ppc64le
restraint-0.1.37-1.git.5.6992e65.el7bkr.aarch64
restraint-0.1.37-1.git.5.6992e65.el8.aarch64

Comment 10 PaulB 2019-01-29 03:22:53 UTC

All(In reply to Jeff Bastian from comment #9)
> Thanks for the quick fix, Bill!  I tested 3 systems and they all worked fine
> (still waiting on a free POWER9 system to test RHEL8):
> 
> restraint-0.1.37-1.git.5.6992e65.el7bkr.ppc64le
> restraint-0.1.37-1.git.5.6992e65.el7bkr.aarch64
> restraint-0.1.37-1.git.5.6992e65.el8.aarch64

All,
 <repos>
   <repo name="myharness" url="http://file.bos.redhat.com/~bpeck/restraint/el8--test/"/>
 </repos>


aarch64 - https://beaker.engineering.redhat.com/jobs/3313417 - PASS
aarch64 - https://beaker.engineering.redhat.com/jobs/3313418 - PASS

ppc64le (Power9) - https://beaker.engineering.redhat.com/jobs/3313473 - PASS

Thank you, bpeck / jbastian

best,
-pbunya

Comment 13 Martin Styk 2019-01-29 12:49:58 UTC

Restraint 0.1.38 has been released.

Comment 14 PaulB 2019-01-29 13:46:53 UTC

(In reply to Martin Styk from comment #13)
> Restraint 0.1.38 has been released.


All,
Testing with updated restraint looks good:
--------
aarch64
--------
https://beaker.engineering.redhat.com/jobs/3314394 - PASS

https://beaker.engineering.redhat.com/jobs/3314395 - PASS


--------
ppc64le
--------
https://beaker.engineering.redhat.com/jobs/3314396 - PASS
http://beaker-archive.host.prod.eng.bos.redhat.com/beaker-logs/2019/01/33143/3314396/6443689/console.log
---<-snip->---
+ yum -y install restraint-rhts
Updating Subscription Management repositories.
Unable to read consumer identity
This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
Last metadata expiration check: 0:00:08 ago on Tue Jan 29 08:23:04 2019.
Dependencies resolved.
================================================================================
 Package            Arch        Version                Repository          Size
================================================================================
Installing:
 restraint-rhts     ppc64le     0.1.38-1.el8bkr        beaker-harness      54 k
Installing dependencies:
 make               ppc64le     1:4.2.1-9.el8          beaker-BaseOS      505 k
 restraint          ppc64le     0.1.38-1.el8bkr        beaker-harness     8.0 M
---<-snip->---


Thank you for the quick fix :)

Best,
-pbunyan

Note You need to log in before you can comment on or make changes to this bug.