Description of problem: 11/03/11 17:23:40 (pid:3354) Setting maximum accepts per cycle 8. 11/03/11 17:23:40 (pid:3354) ****************************************************** 11/03/11 17:23:40 (pid:3354) ** condor_schedd (CONDOR_SCHEDD) STARTING UP 11/03/11 17:23:40 (pid:3354) ** /usr/sbin/condor_schedd 11/03/11 17:23:40 (pid:3354) ** SubsystemInfo: name=SCHEDD type=SCHEDD(5) class=DAEMON(1) 11/03/11 17:23:40 (pid:3354) ** Configuration: subsystem:SCHEDD local:<NONE> class:DAEMON 11/03/11 17:23:40 (pid:3354) ** $CondorVersion: 7.6.5 Oct 31 2011 BuildID: RH-7.6.5-0.5.el5 $ 11/03/11 17:23:40 (pid:3354) ** $CondorPlatform: I686-RedHat_5.7 $ 11/03/11 17:23:40 (pid:3354) ** PID = 3354 11/03/11 17:23:40 (pid:3354) ** Log last touched 11/3 17:14:58 11/03/11 17:23:40 (pid:3354) ****************************************************** 11/03/11 17:23:40 (pid:3354) Using config source: /etc/condor/condor_config 11/03/11 17:23:40 (pid:3354) Using local config sources: 11/03/11 17:23:40 (pid:3354) /etc/condor/config.d/00personal_condor.config 11/03/11 17:23:40 (pid:3354) /etc/condor/config.d/60condor-qmf.config 11/03/11 17:23:40 (pid:3354) /etc/condor/config.d/61aviary.config 11/03/11 17:23:40 (pid:3354) /etc/condor/config.d/99configd.config 11/03/11 17:23:40 (pid:3354) /etc/condor/config.d/99jasan.config 11/03/11 17:23:40 (pid:3354) /var/lib/condor/wallaby_node.config 11/03/11 17:23:40 (pid:3354) DaemonCore: command socket at <10.34.37.173:60872> 11/03/11 17:23:40 (pid:3354) DaemonCore: private command socket at <10.34.37.173:60872> 11/03/11 17:23:40 (pid:3354) Setting maximum accepts per cycle 8. 11/03/11 17:23:40 (pid:3354) ClassAdLogPlugin registration succeeded 11/03/11 17:23:40 (pid:3354) ScheddPlugin registration succeeded 11/03/11 17:23:40 (pid:3354) Successfully loaded plugin: /usr/lib/condor/plugins/MgmtScheddPlugin-plugin.so 11/03/11 17:23:40 (pid:3354) ClassAdLogPlugin registration succeeded 11/03/11 17:23:40 (pid:3354) ScheddPlugin registration succeeded 11/03/11 17:23:40 (pid:3354) Successfully loaded plugin: /usr/lib/condor/plugins/AviaryScheddPlugin-plugin.so 11/03/11 17:23:40 (pid:3354) axis2_ssl_utils_initialize_ctx failed 11/03/11 17:23:40 (pid:3354) SSL/TLS requested but configuration failed Stack dump for process 3354 at timestamp 1320337420 (13 frames) condor_schedd(dprintf_dump_stack+0x4a)[0x81cc96a] condor_schedd[0x81aed76] [0xb44420] /lib/libssl.so.6(SSL_shutdown+0x1a)[0xf8384a] /usr/lib/condor/plugins/AviaryScheddPlugin-plugin.so(axis2_ssl_utils_cleanup_ssl+0x29)[0x359149] /usr/lib/condor/plugins/AviaryScheddPlugin-plugin.so(_ZN6aviary4soap16Axis2SslProviderD0Ev+0x41)[0x359ba1] /usr/lib/condor/plugins/AviaryScheddPlugin-plugin.so(_ZN6aviary9transport21AviaryProviderFactory6createERKSs+0x2c1)[0x357cf1] /usr/lib/condor/plugins/AviaryScheddPlugin-plugin.so(_ZN6aviary3job18AviaryScheddPlugin15earlyInitializeEv+0x66)[0x35a486] condor_schedd(_ZN19ScheddPluginManager15EarlyInitializeEv+0x33)[0x811c9a3] condor_schedd(_Z9main_initiPPc+0xb5)[0x811a0d5] condor_schedd(main+0x1171)[0x8145de1] /lib/libc.so.6(__libc_start_main+0xdc)[0x3cfe9c] condor_schedd[0x80bdb41] Version-Release number of selected component (if applicable): RHEL5 condor-7.6.5-0.5.el5 condor-aviary-7.6.5-0.5.el5 condor-classads-7.6.5-0.5.el5 condor-qmf-7.6.5-0.5.el5 condor-wallaby-base-db-1.16-2.el5 condor-wallaby-client-4.1.2-1.el5 condor-wallaby-tools-4.1.2-1.el5 python-condorutils-1.5-4.el5 RHEL6 condor-7.6.5-0.5.el6 condor-aviary-7.6.5-0.5.el6 condor-classads-7.6.5-0.5.el6 condor-qmf-7.6.5-0.5.el6 condor-wallaby-base-db-1.16-2.el6 condor-wallaby-client-4.1.2-1.el6 condor-wallaby-tools-4.1.2-1.el6 python-condorutils-1.5-4.el6 How reproducible: 100% Steps to Reproduce: [Generate certificates] Use script from Bug 746251 ~]# bash mrg_gen_ssl_certs.sh [List all private keys] ~]# certutil -K -d ~/CA_db/ [Export certificates from database] ~]# cd ~/CA_db/ ~/CA_db]# pk12util -d . -k passwordfile -o ca.p12 -n "NSS Certificate DB:CAnick" ~/CA_db]# pk12util -d . -k passwordfile -o client.p12 -n "NSS Certificate DB:client_..." ~/CA_db]# pk12util -d . -k passwordfile -o serv.p12 -n "NSS Certificate DB:serv_..." [Change certificate format] ~/CA_db]# openssl pkcs12 -in client.p12 -out client.pem -nodes ~/CA_db]# openssl pkcs12 -in serv.p12 -out serv.pem -nodes ~/CA_db]# openssl pkcs12 -in ca.p12 -out ca.pem -nodes [Verify certificates] ~/CA_db]# mkdir ~/ssl ~/CA_db]# cp *.pem ~/ssl ~/CA_db]# cd ~/ssl ~/ssl]# openssl verify -CAfile ./ca.pem serv.pem client.pem ~/ssl]# openssl verify -purpose sslclient -CAfile ./ca.pem serv.pem client.pem ~/ssl]# openssl verify -purpose sslserver -CAfile ./ca.pem serv.pem client.pem ~/ssl]# openssl verify -purpose any -CAfile ./ca.pem serv.pem client.pem [Divide files to cert/key and move to /tmp] # ls /tmp/ssl/ ca_cert.pem ca.pem client_key.pem serv_cert.pem serv.pem ca_key.pem client_cert.pem client.pem serv_key.pem [Test with SSL and without root cert] - /etc/condor/config.d/61aviary.config # SSL support config using OpenSSL default certs dir SCHEDD.AVIARY_SSL = True SCHEDD.AVIARY_SSL_SERVER_CERT = /tmp/ssl/serv_cert.pem SCHEDD.AVIARY_SSL_SERVER_KEY = /tmp/ssl/serv_key.pem SCHEDD.AVIARY_SSL_CA_DIR = SCHEDD.AVIARY_SSL_CA_FILE = QUERY_SERVER.AVIARY_SSL = True QUERY_SERVER.AVIARY_SSL_SERVER_CERT = /tmp/ssl/serv_cert.pem QUERY_SERVER.AVIARY_SSL_SERVER_KEY = /tmp/ssl/serv_key.pem QUERY_SERVER.AVIARY_SSL_CA_DIR = QUERY_SERVER.AVIARY_SSL_CA_FILE = [Run Condor] Actual results: condor_schedd stack dump Expected results: condor_schedd reports problem Additional info:
Created attachment 531736 [details] aviary_query_server dump
Created attachment 531737 [details] condor_schedd dump
Reproduced on RHEL6 x86_64: Configuration: CREATE_CORE_FILES=True ABORT_ON_EXCEPTION=True SCHEDD.AVIARY_SSL = True SCHEDD.AVIARY_SSL_SERVER_CERT = /tmp/ssl/serv_cert.pem SCHEDD.AVIARY_SSL_SERVER_KEY = /tmp/ssl/serv_key.pem SCHEDD.AVIARY_SSL_CA_DIR = SCHEDD.AVIARY_SSL_CA_FILE = QUERY_SERVER.AVIARY_SSL = True QUERY_SERVER.AVIARY_SSL_SERVER_CERT = /tmp/ssl/serv_cert.pem QUERY_SERVER.AVIARY_SSL_SERVER_KEY = /tmp/ssl/serv_key.pem QUERY_SERVER.AVIARY_SSL_CA_DIR = QUERY_SERVER.AVIARY_SSL_CA_FILE = # rpm -qa | grep condor condor-classads-7.6.5-0.5.el6.x86_64 condor-7.6.5-0.5.el6.x86_64 condor-aviary-7.6.5-0.5.el6.x86_64 # cat SchedLog <<<<< truncated output >>>>>> 11/10/11 13:54:54 (pid:31253) ScheddPlugin registration succeeded 11/10/11 13:54:54 (pid:31253) Successfully loaded plugin: /usr/lib64/condor/plugins/AviaryScheddPlugin-plugin.so 11/10/11 13:54:54 (pid:31253) axis2_ssl_utils_initialize_ctx failed 11/10/11 13:54:54 (pid:31253) SSL/TLS requested but configuration failed Stack dump for process 31253 at timestamp 1320929694 (22 frames) condor_schedd(dprintf_dump_stack+0x63)[0x5abba3] condor_schedd[0x5b0a72] /lib64/libpthread.so.0[0x3c2320f490] /usr/lib64/libaxis2_engine.so.0(axis2_phase_free+0x1c)[0x7f5c626e826c] /usr/lib64/libaxis2_engine.so.0(axis2_msg_free+0x7e)[0x7f5c626f6dae] /usr/lib64/libaxis2_engine.so.0(axis2_desc_free+0x59)[0x7f5c626ee319] /usr/lib64/libaxis2_engine.so.0(axis2_op_free+0x1e)[0x7f5c626f013e] /usr/lib64/libaxis2_engine.so.0(axis2_svc_free+0x19e)[0x7f5c626f2f9e] /usr/lib64/libaxis2_engine.so.0(axis2_arch_file_data_free+0xf0)[0x7f5c626ff220] /usr/lib64/libaxis2_engine.so.0(axis2_dep_engine_free+0xae)[0x7f5c626fe3de] /usr/lib64/libaxis2_engine.so.0(axis2_conf_free+0xf0)[0x7f5c626e7450] /usr/lib64/libaxis2_engine.so.0(axis2_conf_ctx_free+0x194)[0x7f5c6270bbf4] /usr/lib64/libaxis2_http_receiver.so.0(+0x1b54)[0x7f5c61c48b54] /usr/lib64/condor/plugins/AviaryScheddPlugin-plugin.so(_ZN6aviary4soap17Axis2SoapProviderD2Ev+0x29)[0x7f5c629424f9] /usr/lib64/condor/plugins/AviaryScheddPlugin-plugin.so(_ZN6aviary4soap16Axis2SslProviderD0Ev+0x9)[0x7f5c629436d9] /usr/lib64/condor/plugins/AviaryScheddPlugin-plugin.so(_ZN6aviary9transport21AviaryProviderFactory6createERKSs+0x27c)[0x7f5c62941fdc] /usr/lib64/condor/plugins/AviaryScheddPlugin-plugin.so(_ZN6aviary3job18AviaryScheddPlugin15earlyInitializeEv+0x62)[0x7f5c62943e82] condor_schedd(_ZN19ScheddPluginManager15EarlyInitializeEv+0x50)[0x4d7e90] condor_schedd(_Z9main_initiPPc+0x7d)[0x47e47d] condor_schedd(main+0x10ef)[0x4de76f] /lib64/libc.so.6(__libc_start_main+0xfd)[0x3c22e1ecdd] condor_schedd[0x47e049] Verified on RHEL6 i386: # rpm -qa | grep condor condor-classads-7.6.5-0.6.el6.i686 condor-7.6.5-0.6.el6.i686 condor-aviary-7.6.5-0.6.el6.i686 # cat SchedLog <<<<< truncated output >>>>>> 11/10/11 13:54:54 (pid:31234) ScheddPlugin registration succeeded 11/10/11 13:54:54 (pid:31234) Successfully loaded plugin: /usr/lib/condor/plugins/AviaryScheddPlugin-plugin.so 11/10/11 13:54:54 (pid:31234) axis2_ssl_utils_initialize_ctx failed 11/10/11 13:54:54 (pid:31234) SSL/TLS requested but configuration failed 11/10/11 13:54:54 (pid:31234) ERROR "Unable to configure AviaryProvider. Exiting..." at line 57 in file /builddir/build/BUILD/condor-7.6.4/src/condor_contrib/aviary/src/AviaryScheddPlugin.cpp Stack dump for process 31234 at timestamp 1320929694 (11 frames) condor_schedd(dprintf_dump_stack+0x44)[0x81a74c4] condor_schedd[0x8192537] [0xb89400] /lib/libc.so.6(abort+0xdd)[0xc023ad] condor_schedd(_EXCEPT_+0xb2)[0x8195832] /usr/lib/condor/plugins/AviaryScheddPlugin-plugin.so(_ZN6aviary3job18AviaryScheddPlugin15earlyInitializeEv+0x22f)[0x14c95f] condor_schedd(_ZN19ScheddPluginManager15EarlyInitializeEv+0x4c)[0x80e41fc] condor_schedd(_Z9main_initiPPc+0x75)[0x8111755] condor_schedd(main+0x13b5)[0x8131245] /lib/libc.so.6(__libc_start_main+0xe6)[0xbecce6] condor_schedd[0x80b5981] Exception about SSL config problem is generated. Output on RHEL5 i386, RHEL5 x86_64 and RHEL6 x86_64 is similar. >>> VERIFIED