Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1593595

Summary: test_fetch_uri segfaults on master
Product: [Retired] Restraint Reporter: Roman Joost <rjoost>
Component: generalAssignee: Roman Joost <rjoost>
Status: CLOSED CURRENTRELEASE QA Contact: Matt Tyson 🤬 <mtyson>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: masterCC: asavkov, bpeck, dcallagh, mtyson, rjoost
Target Milestone: 0.1.36   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-09-17 05:53:06 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Roman Joost 2018-06-21 07:34:42 UTC
Description of problem:

It seems we can't run the tests any more with current HEAD on restraint master (1f636afd2fee504ed13021829f11986f54cb89bd). The test test_fetch_uri fails with:

Thread 1 "test_fetch_uri" received signal SIGSEGV, Segmentation fault.
0x00007ffff64efbe5 in multi_socket (multi=<optimized out>, checkall=<optimized out>, s=<optimized out>, ev_bitmask=<optimized out>, running_handles=0x63764c) at ../../lib/multi.c:2636
2636      return result;

Version-Release number of selected component (if applicable):
1f636afd2fee504ed13021829f11986f54cb89bd

How reproducible:
always

Steps to Reproduce:
1. make check or,
1. make
2. gdb src/test_file_uri
3. run

Actual results:
segfaults

Expected results:
does not segfault

Additional info:
Stacktrace:

(gdb) info stack
#0  0x00007ffff64efb35 in multi_socket () at /lib64/libcurl.so.4
#1  0x702074736f686c61 in  ()
#2  0x303030382074726f in  ()
#3  0x63656e6e6f43203a in  ()
#4  0x666572206e6f6974 in  ()
#5  0x0000000064657375 in  ()
#6  0x00007ffff717e280 in g_source_callback_funcs () at /lib64/libglib-2.0.so.0
#7  0x0000000000619620 in  ()
#8  0x0000000400000001 in  ()
#9  0x0000000000671720 in  ()
#10 0x0000000000671da0 in  ()
#11 0x0000000000612470 in  ()
#12 0x0000000500000002 in  ()
#13 0x0000000000638660 in  ()
#14 0x0000000000638640 in  ()
#15 0x0000000000619620 in  ()
#16 0x0000000000612460 in  ()
#17 0x00007ffff6eb48ad in g_main_dispatch (context=0x638660) at gmain.c:3177
#18 0x00007ffff6eb48ad in g_main_context_dispatch (context=0x638660, context@entry=0x618400) at gmain.c:3830
#19 0x00007ffff6eb4c78 in g_main_context_iterate (context=0x618400, block=block@entry=1, dispatch=dispatch@entry=1, self=<optimized out>) at gmain.c:3903
#20 0x00007ffff6eb4fa2 in g_main_loop_run (loop=0x618540) at gmain.c:4099
#21 0x0000000000402228 in test_fetch_http_nofragment_success () at test_fetch_uri.c:73
#22 0x00007ffff6edbc4a in test_case_run (tc=0x617000) at gtestutils.c:2255
#23 0x00007ffff6edbc4a in g_test_run_suite_internal (suite=suite@entry=0x615e60, path=path@entry=0x0) at gtestutils.c:2339
#24 0x00007ffff6edbb04 in g_test_run_suite_internal (suite=suite@entry=0x615e40, path=path@entry=0x0) at gtestutils.c:2351
#25 0x00007ffff6edbb04 in g_test_run_suite_internal (suite=suite@entry=0x615e20, path=path@entry=0x0) at gtestutils.c:2351
#26 0x00007ffff6edbf02 in g_test_run_suite (suite=0x615e20) at gtestutils.c:2426
#27 0x00007ffff6edbf25 in g_test_run () at gtestutils.c:1692
#28 0x0000000000404128 in main (argc=1, argv=0x7fffffffcf08) at test_fetch_uri.c:624

Comment 2 Roman Joost 2018-06-25 05:19:14 UTC
I can't merge my code from Bug 1590570 unless we've passing tests, might as well fix this.

Comment 3 Roman Joost 2018-06-25 05:32:11 UTC
I've checked our slaves which have:

glibc-2.25-12.fc26.src.rpm
glib2-2.52.3-2.fc26.src.rpm
curl-7.53.1-13.fc26.src.rpm

and on a newly provisioned F26 machine we have:

glibc-2.25-13.fc26.src.rpm
glib2-2.52.3-2.fc26.src.rpm
curl-7.53.1-16.fc26.src.rpm

Comment 4 Roman Joost 2018-06-26 05:34:33 UTC
I've now confirmed that building it from source on our Fedora 26 Jenkins instance works. I've downgraded all obvious dependencies of restraint on a newly provisioned Fedora 26 instance in Beaker to match up what I have identified with the Jenkins instance. Yet it still segfaults in libcurl (curl_multi_socket_action).

I've taken a different approach and grabbed the sources in distgit for the two builds from Fedora:

curl-7.53.1-13.fc26
https://koji.fedoraproject.org/koji/buildinfo?buildID=1006496

curl-7.53.1-16.fc26
https://koji.fedoraproject.org/koji/buildinfo?buildID=1058139

and prepared a static build from the third-party folder in restraint with these sources. The curl sources were fully patched before. The test runs without a segfault.

Now I'm beginning to think that the data being handed to libcurl through the g_main_loop_run is getting somehow corrupted. I first thought that our glib version in the third-party folder was too old, but it just had recently been updated by Matt.

Perhaps next attempt is to try watching the data with gdb and see if it indeed gets corrupted and by what.

Comment 5 Roman Joost 2018-06-27 06:12:53 UTC
Heh so that one was a bit of a layer cake. Matt found the point in the source code, where we allocated memory for a cURL error buffer on the stack. This buffer however needs to stay around for the livetime of the connection in order for cURL to write to it.

For this particular test it did actually try to write the error, yet at that point the buffer was already gone.

The actual cURL error was:

* SMB shares are not supported in file: URLs.

which oddly enough we don't use. All we have is a file URL pointing to a local directory:

g_strdup_printf("file:///%s/test-data/http-remote/fetch_http.tgz#restraint/sanity/fetch_git", cwd)

What that becomes however is: file:////home/rjoost/works/restraint/src/test-data/http-remote/fetch_http.tgz#restraint/sanity/fetch_git
(Note the additional slash in the authority)

Also related to this is: https://github.com/curl/curl/commit/695e96b3d5b9e6ae4832af5f8e99e07de960bf33

Comment 6 Matt Tyson 🤬 2018-07-05 04:10:44 UTC
Reverting the patch and running the test suite shows us the crash.  With the patch applied we don't get the crash any more.

Comment 7 Dan Callaghan 2018-09-17 05:53:06 UTC
Restraint 0.1.36 was released on 24 August.