Bug 1358845

Summary: nghttpx crashes on armv7hl while running HTTP/2 curl tests
Product: [Fedora] Fedora Reporter: Kamil Dudka <kdudka>
Component: nghttp2Assignee: Kamil Dudka <kdudka>
Status: CLOSED CURRENTRELEASE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: medium Docs Contact:
Priority: medium    
Version: 25CC: kdudka, pwhalen
Target Milestone: ---   
Target Release: ---   
Hardware: armv7hl   
OS: Linux   
Whiteboard:
Fixed In Version: nghttp2-1.13.0-2.fc25 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-07-29 13:56:05 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1360319    
Bug Blocks: 245418    
Attachments:
Description Flags
[PATCH] nghttpx: avoid using std::function to fix crash on armv7hl none

Description Kamil Dudka 2016-07-21 15:19:20 UTC
Version-Release number of selected component (if applicable):
nghttp2-1.13.0-1.fc25.armv7hl


Actual results:
[INFO] shrpx_config.cc:2881 Resolving backend address
[INFO] shrpx_config.cc:2891 Host-path pattern: group 0: '/'
[INFO] shrpx_config.cc:2894 group 0 -> 127.0.0.1:8990, proto=http/1.1
[INFO] shrpx_config.cc:2910 Catch-all pattern is group 0
[INFO] shrpx_config.cc:3002 Address resolution for 127.0.0.1 succeeded: 127.0.0.1
[INFO] shrpx_config.cc:2958 Resolved backend address: 127.0.0.1:8990 -> 127.0.0.1:8990
[NOTICE] Listening on 0.0.0.0:9015
[NOTICE] Listening on [::]:9015
[NOTICE] Worker process [29578] spawned
[INFO] shrpx_worker.cc:265 number of http/1.1 backend: 1, number of h2 backend: 0
[INFO] shrpx_worker_process.cc:528 Entering event loop
[INFO] shrpx_connection_handler.cc:365 [LISTEN:0xbefab428] Accepted connection. fd=11
[INFO] shrpx_client_handler.cc:70 [CLIENT_HANDLER:0x80e91750] Time out
[INFO] shrpx_client_handler.cc:449 [CLIENT_HANDLER:0x80e91750] Deleting
[INFO] shrpx_client_handler.cc:472 [CLIENT_HANDLER:0x80e91750] Deleted
[INFO] shrpx_connection_handler.cc:365 [LISTEN:0xbefab428] Accepted connection. fd=11
[NOTICE] Worker process: [29578] exited abnormally with status 8b; exit status 0; signal Segmentation fault(11)
[NOTICE] Shutdown momentarily


Additional info:
http://koji.fedoraproject.org/koji/taskinfo?taskID=14968582
http://koji.fedoraproject.org/koji/taskinfo?taskID=14968582

Comment 1 Kamil Dudka 2016-07-25 15:30:30 UTC
Reproduced on an f24 box:

# rpm -qa libev\* \*nghttp\* | sort -V
libev-4.20-2.fc24.armv7hl
libev-debuginfo-4.20-2.fc24.armv7hl
libev-devel-4.20-2.fc24.armv7hl
libnghttp2-1.13.0-1.fc24.armv7hl
libnghttp2-devel-1.13.0-1.fc24.armv7hl
nghttp2-1.13.0-1.fc24.armv7hl
nghttp2-debuginfo-1.13.0-1.fc24.armv7hl

(gdb) bt
#0  0x7f62ebc4 in std::__invoke_impl<int, int (shrpx::ClientHandler::* const&)(), shrpx::ClientHandler&> (__t=..., __f=<optimized out>) at /usr/include/c++/6.1.1/functional:226
#1  std::__invoke<int (shrpx::ClientHandler::* const&)(), shrpx::ClientHandler&> (__fn=<optimized out>) at /usr/include/c++/6.1.1/functional:260
#2  std::_Mem_fn_base<int (shrpx::ClientHandler::*)(), true>::operator()<shrpx::ClientHandler&> (this=<optimized out>) at /usr/include/c++/6.1.1/functional:613
#3  std::_Function_handler<int (shrpx::ClientHandler&), int (shrpx::ClientHandler::*)()>::_M_invoke(std::_Any_data const&, shrpx::ClientHandler&) (__functor=..., __args#0=...) at /usr/include/c++/6.1.1/functional:1788
#4  0x7f6295b0 in std::function<int (shrpx::ClientHandler&)>::operator()(shrpx::ClientHandler&) const (__args#0=..., this=0x7fadce8c) at /usr/include/c++/6.1.1/functional:2136
#5  shrpx::ClientHandler::do_read (this=0x7fadcc60) at shrpx_client_handler.cc:615
#6  shrpx::(anonymous namespace)::readcb (loop=<optimized out>, w=<optimized out>, revents=<optimized out>) at shrpx_client_handler.cc:94
#7  0xb6e9893c in ev_invoke_pending (loop=0xb6eb0358 <default_loop_struct>) at ev.c:3155
#8  0xb6e9bc18 in ev_run (loop=0xb6eb0358 <default_loop_struct>, loop@entry=0x0, flags=-1226195944, flags@entry=0) at ev.c:3555
#9  0x7f618f30 in shrpx::worker_process_event_loop (wpconf=0x0) at shrpx_worker_process.cc:537
#10 0x7f5bf524 in shrpx::(anonymous namespace)::fork_worker_process (ssv=0xbeaf1c8c) at shrpx.cc:884
#11 shrpx::(anonymous namespace)::event_loop () at shrpx.cc:966
#12 0x7f5b9678 in shrpx::main (argc=6, argv=0xbeaf45b4) at shrpx.cc:3113
#13 0x7f5c1b58 in std::function<int (int, char**)>::operator()(int, char**) const (__args#1=<optimized out>, __args#0=<optimized out>, this=<optimized out>) at /usr/include/c++/6.1.1/functional:2136
#14 nghttp2::run_app(std::function<int (int, char**)>, int, char**) (app=..., argc=<optimized out>, argv=<optimized out>) at template.h:542
#15 0x7f5b7c90 in main (argc=<optimized out>, argv=<optimized out>) at shrpx.cc:3126

If I rebuild/reinstall libev from SRPM locally (the same NVR), nghttpx stops crashing but HTTP/2 curl's tests fail anyway after a while of hanging.

Comment 2 Kamil Dudka 2016-07-25 15:44:04 UTC
> If I rebuild/reinstall libev from SRPM locally (the same NVR), nghttpx stops
> crashing but HTTP/2 curl's tests fail anyway after a while of hanging.

Actually, only the first test hangs whereas the subsequent tests still crash the nghttpx worker.  It is reproducible also with libev-4.22-1.fc25 rebuilt for f24.

Here is another backtrace caught with the updated libev:

(gdb) bt
#0  0x00000000 in ?? ()
#1  std::function<int (shrpx::ClientHandler&)>::operator()(shrpx::ClientHandler&) const (__args#0=..., this=0x81188e8c) at /usr/include/c++/6.1.1/functional:2136
#2  shrpx::ClientHandler::do_read (this=0x81188c60) at shrpx_client_handler.cc:615
#3  shrpx::(anonymous namespace)::readcb (loop=<optimized out>, w=<optimized out>, revents=<optimized out>) at shrpx_client_handler.cc:94
#4  ev_invoke_pending (loop=0xb6f51358 <default_loop_struct>) at ev.c:3288
#5  ev_run (loop=0xb6f51358 <default_loop_struct>, loop@entry=0x0, flags=-1225536504, flags@entry=0) at ev.c:3688
#6  shrpx::worker_process_event_loop (wpconf=0x0) at shrpx_worker_process.cc:537
#7  shrpx::(anonymous namespace)::fork_worker_process (ssv=0xbeb99c8c) at shrpx.cc:884
#8  shrpx::(anonymous namespace)::event_loop () at shrpx.cc:966
#9  shrpx::main (argc=6, argv=0xbeb9c5b4) at shrpx.cc:3113
#10 std::function<int (int, char**)>::operator()(int, char**) const (__args#1=<optimized out>, __args#0=<optimized out>, this=<optimized out>) at /usr/include/c++/6.1.1/functional:2136
#11 nghttp2::run_app(std::function<int (int, char**)>, int, char**) (app=..., argc=<optimized out>, argv=<optimized out>) at template.h:542
#12 main (argc=<optimized out>, argv=<optimized out>) at shrpx.cc:3126

Comment 3 Kamil Dudka 2016-07-25 17:38:25 UTC
Rebuilding nghttp2 with -O1 in CXXFLAGS prevents nghttpx from crashing.

Comment 4 Kamil Dudka 2016-07-25 22:28:32 UTC
-- a/src/Makefile.am
+++ b/src/Makefile.am
@@ -156,8 +156,12 @@ endif # HAVE_MRUBY
 noinst_LIBRARIES = libnghttpx.a
 libnghttpx_a_SOURCES = ${NGHTTPX_SRCS}
 libnghttpx_a_CPPFLAGS = ${AM_CPPFLAGS}

+# needed for shrpx_client_handler.cc and shrpx_http_downstream_connection.cc
+# on armv7hl -- see https://bugzilla.redhat.com/1358845 for details
+libnghttpx_a_CXXFLAGS = -fno-strict-aliasing
+
 nghttpx_SOURCES = shrpx.cc shrpx.h
 nghttpx_CPPFLAGS = ${libnghttpx_a_CPPFLAGS}
 nghttpx_LDADD = libnghttpx.a ${LDADD}

Comment 5 Jan Kurik 2016-07-26 04:30:14 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 25 development cycle.
Changing version to '25'.

Comment 6 Kamil Dudka 2016-07-26 11:11:01 UTC
Created attachment 1184185 [details]
[PATCH] nghttpx: avoid using std::function to fix crash on armv7hl

There seems to be some problem with nghttp's use of std::function.  The attached patch makes the crashes go away.  I am not familiar with std::function myself, so it is not clear to me whether it is a bug of nghttp or a gcc bug.

Comment 7 Kamil Dudka 2016-07-26 12:30:14 UTC
Reported as gcc bug #1360319 with minimal example: attachment #1184246 [details]

Comment 8 Kamil Dudka 2016-07-26 15:07:48 UTC
Reported to nghttp2 upstream: https://github.com/nghttp2/nghttp2/issues/635

Comment 9 Kamil Dudka 2016-07-29 13:56:05 UTC
curl now builds on armv7hl without any workaround in place:

http://koji.fedoraproject.org/koji/buildinfo?buildID=785581