Simply invoking jobquery.py getJobStatus on the aviary schedd endpoint causes a crash: Stack dump for process 17358 at timestamp 1316101931 (27 frames) condor_schedd(dprintf_dump_stack+0xc7)[0x8256fea] condor_schedd[0x823025e] [0x335400] [0x335416] /lib/libc.so.6(gsignal+0x51)[0xe012f1] /lib/libc.so.6(abort+0x17e)[0xe02d5e] /lib/libc.so.6(+0x21651d)[0xe3d51d] /lib/libc.so.6(+0x21c871)[0xe43871] /usr/lib/libaxutil.so.0(axutil_allocator_free_impl+0x1d)[0x251553] /usr/lib/libwsf_cpp_msg_recv.so.0(+0x285f)[0x95185f] /usr/lib/libaxis2_engine.so.0(axis2_msg_recv_invoke_business_logic+0x29)[0x4e1554] /usr/lib/libaxis2_engine.so.0[0x4e133d] /usr/lib/libaxis2_engine.so.0(axis2_msg_recv_receive+0x29)[0x4e15b0] /usr/lib/libaxis2_engine.so.0(axis2_engine_receive+0x551)[0x4d317d] /usr/lib/libaxis2_engine.so.0(axis2_http_transport_utils_process_http_post_request+0x119c)[0x5125e5] /usr/lib/libaxis2_http_common.so.0(axis2_http_worker_process_request+0x1c46)[0x454684] /home/pmackinn/personal-condor/release_dir/uw/libexec/AviaryScheddPlugin-plugin.so(_ZN6aviary4soap17Axis2SoapProvider12invokeWorkerEP15axutil_thread_tPv+0x19c)[0x1fb7e6] /home/pmackinn/personal-condor/release_dir/uw/libexec/AviaryScheddPlugin-plugin.so(_ZN6aviary4soap17Axis2SoapProvider14processRequestERSs+0x17e)[0x1fb5e2] /home/pmackinn/personal-condor/release_dir/uw/libexec/AviaryScheddPlugin-plugin.so(_ZN6aviary3job18AviaryScheddPlugin21HandleTransportSocketEP6Stream+0x41)[0x1fd493] condor_schedd(_ZN10DaemonCore24CallSocketHandler_workerEibP6Stream+0x24c)[0x8176db4] condor_schedd(_ZN10DaemonCore35CallSocketHandler_worker_demarshallEPv+0x3b)[0x8176b57] condor_schedd(_ZN13CondorThreads8pool_addEPFvPvES0_PiPKc+0x29)[0x821825b] condor_schedd(_ZN10DaemonCore17CallSocketHandlerERib+0x296)[0x8176aee] condor_schedd(_ZN10DaemonCore6DriverEv+0x1807)[0x8176701] condor_schedd(main+0x1dfb)[0x816e4e8] /lib/libc.so.6(__libc_start_main+0xe6)[0xdece36] condor_schedd[0x8109801]
The problem is that the URL is parsed by Axis2/C as a key to load the shared lib containing the impl. This step fails if there is a mismatch and then further down the stack Axis2/C blithely tries to free what was never there. Will try to mitigate this in Aviary code above Axis2/C.
Decided to address the root problem. Patch upstream (WS02) at: https://wso2.org/jira/browse/WSFCPP-138
Note this will be fixed by applying the upstream patch to the wso2-wsf-cpp pkg build.
Included patch in latest wso2 spin
Technical note added. If any revisions are required, please edit the "Technical Notes" field accordingly. All revisions will be proofread by the Engineering Content Services team. New Contents: Cause: Invoking the wrong endpoint URL in Aviary. Consequence: The WSO2/Axis2C engine used by Aviary fails to load an implementation library and crashes the process. Fix: A statically allocated fault string was incorrectly freed in a WSO2 code path. This was changed to ensure that only a dynamically allocated string would be freed in the same code path. Result: Process containing Aviary doesn't crash and Aviary client receives a SOAP fault for wrong endpoint invocation.
Technical note updated. If any revisions are required, please edit the "Technical Notes" field accordingly. All revisions will be proofread by the Engineering Content Services team. Diffed Contents: @@ -1,7 +1 @@ -Cause: Invoking the wrong endpoint URL in Aviary. +Cause: Invoking the wrong endpoint URL in Aviary. Consequence: The WSO2/Axis2C engine used by Aviary fails to load an implementation library and crashes the process. Fix: A statically allocated fault string was incorrectly freed in a WSO2 code path. This was changed to ensure that only a dynamically allocated string would be freed in the same code path. Result: Process containing Aviary doesn't crash and Aviary client receives a SOAP fault for wrong endpoint invocation.- -Consequence: The WSO2/Axis2C engine used by Aviary fails to load an implementation library and crashes the process. - -Fix: A statically allocated fault string was incorrectly freed in a WSO2 code path. This was changed to ensure that only a dynamically allocated string would be freed in the same code path. - -Result: Process containing Aviary doesn't crash and Aviary client receives a SOAP fault for wrong endpoint invocation.
Technical note updated. If any revisions are required, please edit the "Technical Notes" field accordingly. All revisions will be proofread by the Engineering Content Services team. Diffed Contents: @@ -1 +1,7 @@ -Cause: Invoking the wrong endpoint URL in Aviary. Consequence: The WSO2/Axis2C engine used by Aviary fails to load an implementation library and crashes the process. Fix: A statically allocated fault string was incorrectly freed in a WSO2 code path. This was changed to ensure that only a dynamically allocated string would be freed in the same code path. Result: Process containing Aviary doesn't crash and Aviary client receives a SOAP fault for wrong endpoint invocation.+Cause: Invoking the wrong endpoint URL in Aviary. + +Consequence: The WSO2/Axis2C engine used by Aviary fails to load an implementation library and crashes the process. + +Fix: A statically allocated fault string was incorrectly freed in a WSO2 code path. This was changed to ensure that only a dynamically allocated string would be freed in the same code path. + +Result: Process containing Aviary doesn't crash and Aviary client receives a SOAP fault for wrong endpoint invocation.
Successfully reproduced with wso2-axis2-2.1.0-3 Stack dump for process 21347 at timestamp 1320153189 (22 frames) condor_schedd(dprintf_dump_stack+0x56)[0x5d7346] condor_schedd[0x5adc92] /lib64/libpthread.so.0[0x354c60eb10] /lib64/libc.so.6(abort+0x28f)[0x354ba31e8f] /lib64/libc.so.6[0x354ba6a99b] /lib64/libc.so.6(cfree+0x166)[0x354ba729d6] /usr/lib64/libwsf_cpp_msg_recv.so.0[0x2ad609893eab] /usr/lib64/libaxis2_engine.so.0[0x2ad608363ff1] /usr/lib64/libaxis2_engine.so.0(axis2_engine_receive+0x399)[0x2ad60835aa59] /usr/lib64/libaxis2_engine.so.0(axis2_http_transport_utils_process_http_post_request+0x692)[0x2ad608387c32] /usr/lib64/libaxis2_http_common.so.0(axis2_http_worker_process_request+0x1d2a)[0x2ad608c0e11a] /usr/lib64/condor/plugins/AviaryScheddPlugin-plugin.so(_ZN6aviary4soap17Axis2SoapProvider16invokeHttpWorkerEP15axutil_thread_tPv+0xa3)[0x2ad60811ff03] /usr/lib64/condor/plugins/AviaryScheddPlugin-plugin.so(_ZN6aviary4soap17Axis2SoapProvider18processHttpRequestERSs+0x85)[0x2ad608120205] /usr/lib64/condor/plugins/AviaryScheddPlugin-plugin.so(_ZN6aviary3job18AviaryScheddPlugin21HandleTransportSocketEP6Stream+0x34)[0x2ad608121054] condor_schedd(_ZN10DaemonCore24CallSocketHandler_workerEibP6Stream+0x498)[0x4f9fb8] condor_schedd(_ZN10DaemonCore35CallSocketHandler_worker_demarshallEPv+0x1a)[0x4fa45a] condor_schedd(_ZN13CondorThreads8pool_addEPFvPvES0_PiPKc+0x38)[0x5aaed8] condor_schedd(_ZN10DaemonCore17CallSocketHandlerERib+0x149)[0x4f2b39] condor_schedd(_ZN10DaemonCore6DriverEv+0x1bb5)[0x4f4da5] condor_schedd(main+0xe60)[0x508d00] /lib64/libc.so.6(__libc_start_main+0xf4)[0x354ba1d994] condor_schedd[0x486629]
Tested with: wso2-axis2-2.1.0-5 Tested on: RHEL5 x86_64, i386 RHEL6 x86_64, i386 No stack dump of scheduler. >>> VERIFIED
Technical note updated. If any revisions are required, please edit the "Technical Notes" field accordingly. All revisions will be proofread by the Engineering Content Services team. Diffed Contents: @@ -1,7 +1 @@ -Cause: Invoking the wrong endpoint URL in Aviary. +Previously, a statically allocated fault string was incorrectly freed in a WSO2 code path. When a wrong endpoint URL was invoked in the Aviary web service, The WSO2/Axis2C engine used by Aviary failed to load an implementation library and terminated the process unexpectedly. With this update, only a dynamically allocated string is freed in the same code path, the process containing Aviary no longer crashes in the described scenario, and Aviary clients now receive SOAP errors for wrong endpoint invocation.- -Consequence: The WSO2/Axis2C engine used by Aviary fails to load an implementation library and crashes the process. - -Fix: A statically allocated fault string was incorrectly freed in a WSO2 code path. This was changed to ensure that only a dynamically allocated string would be freed in the same code path. - -Result: Process containing Aviary doesn't crash and Aviary client receives a SOAP fault for wrong endpoint invocation.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHEA-2012-0045.html