Bug 751278

Summary: condor_schedd + AviaryScheddPlugin Stack dump
Product: Red Hat Enterprise MRG Reporter: Stanislav Graf <sgraf>
Component: condor-aviaryAssignee: Pete MacKinnon <pmackinn>
Status: CLOSED ERRATA QA Contact: Daniel Horák <dahorak>
Severity: high Docs Contact:
Priority: high    
Version: 2.1CC: dahorak, iboverma, jneedle, ltoscano, matt, mkudlej, tstclair
Target Milestone: 2.1   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: condor-7.6.5-0.6 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2012-01-27 19:12:46 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 733677, 814386    
Attachments:
Description Flags
aviary_query_server dump
none
condor_schedd dump none

Description Stanislav Graf 2011-11-04 06:57:43 UTC 
Description of problem:

11/03/11 17:23:40 (pid:3354) Setting maximum accepts per cycle 8.
11/03/11 17:23:40 (pid:3354) ******************************************************
11/03/11 17:23:40 (pid:3354) ** condor_schedd (CONDOR_SCHEDD) STARTING UP
11/03/11 17:23:40 (pid:3354) ** /usr/sbin/condor_schedd
11/03/11 17:23:40 (pid:3354) ** SubsystemInfo: name=SCHEDD type=SCHEDD(5) class=DAEMON(1)
11/03/11 17:23:40 (pid:3354) ** Configuration: subsystem:SCHEDD local:<NONE> class:DAEMON
11/03/11 17:23:40 (pid:3354) ** $CondorVersion: 7.6.5 Oct 31 2011 BuildID: RH-7.6.5-0.5.el5 $
11/03/11 17:23:40 (pid:3354) ** $CondorPlatform: I686-RedHat_5.7 $
11/03/11 17:23:40 (pid:3354) ** PID = 3354
11/03/11 17:23:40 (pid:3354) ** Log last touched 11/3 17:14:58
11/03/11 17:23:40 (pid:3354) ******************************************************
11/03/11 17:23:40 (pid:3354) Using config source: /etc/condor/condor_config
11/03/11 17:23:40 (pid:3354) Using local config sources:
11/03/11 17:23:40 (pid:3354)    /etc/condor/config.d/00personal_condor.config
11/03/11 17:23:40 (pid:3354)    /etc/condor/config.d/60condor-qmf.config
11/03/11 17:23:40 (pid:3354)    /etc/condor/config.d/61aviary.config
11/03/11 17:23:40 (pid:3354)    /etc/condor/config.d/99configd.config
11/03/11 17:23:40 (pid:3354)    /etc/condor/config.d/99jasan.config
11/03/11 17:23:40 (pid:3354)    /var/lib/condor/wallaby_node.config
11/03/11 17:23:40 (pid:3354) DaemonCore: command socket at <10.34.37.173:60872>
11/03/11 17:23:40 (pid:3354) DaemonCore: private command socket at <10.34.37.173:60872>
11/03/11 17:23:40 (pid:3354) Setting maximum accepts per cycle 8.
11/03/11 17:23:40 (pid:3354) ClassAdLogPlugin registration succeeded
11/03/11 17:23:40 (pid:3354) ScheddPlugin registration succeeded
11/03/11 17:23:40 (pid:3354) Successfully loaded plugin: /usr/lib/condor/plugins/MgmtScheddPlugin-plugin.so
11/03/11 17:23:40 (pid:3354) ClassAdLogPlugin registration succeeded
11/03/11 17:23:40 (pid:3354) ScheddPlugin registration succeeded
11/03/11 17:23:40 (pid:3354) Successfully loaded plugin: /usr/lib/condor/plugins/AviaryScheddPlugin-plugin.so
11/03/11 17:23:40 (pid:3354) axis2_ssl_utils_initialize_ctx failed
11/03/11 17:23:40 (pid:3354) SSL/TLS requested but configuration failed
Stack dump for process 3354 at timestamp 1320337420 (13 frames)
condor_schedd(dprintf_dump_stack+0x4a)[0x81cc96a]
condor_schedd[0x81aed76]
[0xb44420]
/lib/libssl.so.6(SSL_shutdown+0x1a)[0xf8384a]
/usr/lib/condor/plugins/AviaryScheddPlugin-plugin.so(axis2_ssl_utils_cleanup_ssl+0x29)[0x359149]
/usr/lib/condor/plugins/AviaryScheddPlugin-plugin.so(_ZN6aviary4soap16Axis2SslProviderD0Ev+0x41)[0x359ba1]
/usr/lib/condor/plugins/AviaryScheddPlugin-plugin.so(_ZN6aviary9transport21AviaryProviderFactory6createERKSs+0x2c1)[0x357cf1]
/usr/lib/condor/plugins/AviaryScheddPlugin-plugin.so(_ZN6aviary3job18AviaryScheddPlugin15earlyInitializeEv+0x66)[0x35a486]
condor_schedd(_ZN19ScheddPluginManager15EarlyInitializeEv+0x33)[0x811c9a3]
condor_schedd(_Z9main_initiPPc+0xb5)[0x811a0d5]
condor_schedd(main+0x1171)[0x8145de1]
/lib/libc.so.6(__libc_start_main+0xdc)[0x3cfe9c]
condor_schedd[0x80bdb41] 

Version-Release number of selected component (if applicable):
RHEL5
condor-7.6.5-0.5.el5
condor-aviary-7.6.5-0.5.el5
condor-classads-7.6.5-0.5.el5
condor-qmf-7.6.5-0.5.el5
condor-wallaby-base-db-1.16-2.el5
condor-wallaby-client-4.1.2-1.el5
condor-wallaby-tools-4.1.2-1.el5
python-condorutils-1.5-4.el5

RHEL6
condor-7.6.5-0.5.el6
condor-aviary-7.6.5-0.5.el6
condor-classads-7.6.5-0.5.el6
condor-qmf-7.6.5-0.5.el6
condor-wallaby-base-db-1.16-2.el6
condor-wallaby-client-4.1.2-1.el6
condor-wallaby-tools-4.1.2-1.el6
python-condorutils-1.5-4.el6


How reproducible:
100%

Steps to Reproduce:
[Generate certificates]
Use script from Bug 746251
~]# bash mrg_gen_ssl_certs.sh

[List all private keys]
~]# certutil -K -d ~/CA_db/

[Export certificates from database]
~]# cd ~/CA_db/
~/CA_db]# pk12util -d . -k passwordfile -o ca.p12     -n "NSS Certificate DB:CAnick"
~/CA_db]# pk12util -d . -k passwordfile -o client.p12 -n "NSS Certificate DB:client_..."
~/CA_db]# pk12util -d . -k passwordfile -o serv.p12   -n "NSS Certificate DB:serv_..."

[Change certificate format]
~/CA_db]# openssl pkcs12 -in client.p12 -out client.pem -nodes
~/CA_db]# openssl pkcs12 -in serv.p12   -out serv.pem   -nodes
~/CA_db]# openssl pkcs12 -in ca.p12     -out ca.pem     -nodes

[Verify certificates]
~/CA_db]# mkdir ~/ssl
~/CA_db]# cp *.pem ~/ssl
~/CA_db]# cd ~/ssl
~/ssl]# openssl verify -CAfile                    ./ca.pem serv.pem client.pem
~/ssl]# openssl verify -purpose sslclient -CAfile ./ca.pem serv.pem client.pem
~/ssl]# openssl verify -purpose sslserver -CAfile ./ca.pem serv.pem client.pem
~/ssl]# openssl verify -purpose any       -CAfile ./ca.pem serv.pem client.pem

[Divide files to cert/key and move to /tmp]
# ls /tmp/ssl/
ca_cert.pem  ca.pem           client_key.pem  serv_cert.pem  serv.pem
ca_key.pem   client_cert.pem  client.pem      serv_key.pem

[Test with SSL and without root cert]
- /etc/condor/config.d/61aviary.config
# SSL support config using OpenSSL default certs dir
SCHEDD.AVIARY_SSL = True
SCHEDD.AVIARY_SSL_SERVER_CERT = /tmp/ssl/serv_cert.pem
SCHEDD.AVIARY_SSL_SERVER_KEY = /tmp/ssl/serv_key.pem
SCHEDD.AVIARY_SSL_CA_DIR =
SCHEDD.AVIARY_SSL_CA_FILE =
QUERY_SERVER.AVIARY_SSL = True
QUERY_SERVER.AVIARY_SSL_SERVER_CERT = /tmp/ssl/serv_cert.pem
QUERY_SERVER.AVIARY_SSL_SERVER_KEY = /tmp/ssl/serv_key.pem
QUERY_SERVER.AVIARY_SSL_CA_DIR =
QUERY_SERVER.AVIARY_SSL_CA_FILE = 

[Run Condor]
  
Actual results:
condor_schedd stack dump

Expected results:
condor_schedd reports problem

Additional info:
Comment 3 Stanislav Graf 2011-11-04 10:08:54 UTC 
Created attachment 531736 [details]
aviary_query_server dump
Comment 4 Stanislav Graf 2011-11-04 10:09:25 UTC 
Created attachment 531737 [details]
condor_schedd dump
Comment 7 Daniel Horák 2011-11-10 13:09:47 UTC 
Reproduced on RHEL6 x86_64:

Configuration: 
CREATE_CORE_FILES=True
ABORT_ON_EXCEPTION=True
SCHEDD.AVIARY_SSL = True
SCHEDD.AVIARY_SSL_SERVER_CERT = /tmp/ssl/serv_cert.pem
SCHEDD.AVIARY_SSL_SERVER_KEY = /tmp/ssl/serv_key.pem
SCHEDD.AVIARY_SSL_CA_DIR = 
SCHEDD.AVIARY_SSL_CA_FILE = 
QUERY_SERVER.AVIARY_SSL = True
QUERY_SERVER.AVIARY_SSL_SERVER_CERT = /tmp/ssl/serv_cert.pem
QUERY_SERVER.AVIARY_SSL_SERVER_KEY = /tmp/ssl/serv_key.pem
QUERY_SERVER.AVIARY_SSL_CA_DIR = 
QUERY_SERVER.AVIARY_SSL_CA_FILE = 


# rpm -qa | grep condor
  condor-classads-7.6.5-0.5.el6.x86_64
  condor-7.6.5-0.5.el6.x86_64
  condor-aviary-7.6.5-0.5.el6.x86_64

# cat SchedLog 
        <<<<< truncated output >>>>>>
  11/10/11 13:54:54 (pid:31253) ScheddPlugin registration succeeded
  11/10/11 13:54:54 (pid:31253) Successfully loaded plugin: /usr/lib64/condor/plugins/AviaryScheddPlugin-plugin.so
  11/10/11 13:54:54 (pid:31253) axis2_ssl_utils_initialize_ctx failed
  11/10/11 13:54:54 (pid:31253) SSL/TLS requested but configuration failed
  Stack dump for process 31253 at timestamp 1320929694 (22 frames)
  condor_schedd(dprintf_dump_stack+0x63)[0x5abba3]
  condor_schedd[0x5b0a72]
  /lib64/libpthread.so.0[0x3c2320f490]
  /usr/lib64/libaxis2_engine.so.0(axis2_phase_free+0x1c)[0x7f5c626e826c]
  /usr/lib64/libaxis2_engine.so.0(axis2_msg_free+0x7e)[0x7f5c626f6dae]
  /usr/lib64/libaxis2_engine.so.0(axis2_desc_free+0x59)[0x7f5c626ee319]
  /usr/lib64/libaxis2_engine.so.0(axis2_op_free+0x1e)[0x7f5c626f013e]
  /usr/lib64/libaxis2_engine.so.0(axis2_svc_free+0x19e)[0x7f5c626f2f9e]
  /usr/lib64/libaxis2_engine.so.0(axis2_arch_file_data_free+0xf0)[0x7f5c626ff220]
  /usr/lib64/libaxis2_engine.so.0(axis2_dep_engine_free+0xae)[0x7f5c626fe3de]
  /usr/lib64/libaxis2_engine.so.0(axis2_conf_free+0xf0)[0x7f5c626e7450]
  /usr/lib64/libaxis2_engine.so.0(axis2_conf_ctx_free+0x194)[0x7f5c6270bbf4]
  /usr/lib64/libaxis2_http_receiver.so.0(+0x1b54)[0x7f5c61c48b54]
  /usr/lib64/condor/plugins/AviaryScheddPlugin-plugin.so(_ZN6aviary4soap17Axis2SoapProviderD2Ev+0x29)[0x7f5c629424f9]
  /usr/lib64/condor/plugins/AviaryScheddPlugin-plugin.so(_ZN6aviary4soap16Axis2SslProviderD0Ev+0x9)[0x7f5c629436d9]
  /usr/lib64/condor/plugins/AviaryScheddPlugin-plugin.so(_ZN6aviary9transport21AviaryProviderFactory6createERKSs+0x27c)[0x7f5c62941fdc]
  /usr/lib64/condor/plugins/AviaryScheddPlugin-plugin.so(_ZN6aviary3job18AviaryScheddPlugin15earlyInitializeEv+0x62)[0x7f5c62943e82]
  condor_schedd(_ZN19ScheddPluginManager15EarlyInitializeEv+0x50)[0x4d7e90]
  condor_schedd(_Z9main_initiPPc+0x7d)[0x47e47d]
  condor_schedd(main+0x10ef)[0x4de76f]
  /lib64/libc.so.6(__libc_start_main+0xfd)[0x3c22e1ecdd]
  condor_schedd[0x47e049]



Verified on RHEL6 i386:
# rpm -qa | grep condor
  condor-classads-7.6.5-0.6.el6.i686
  condor-7.6.5-0.6.el6.i686
  condor-aviary-7.6.5-0.6.el6.i686

# cat SchedLog 
        <<<<< truncated output >>>>>>
  11/10/11 13:54:54 (pid:31234) ScheddPlugin registration succeeded
  11/10/11 13:54:54 (pid:31234) Successfully loaded plugin: /usr/lib/condor/plugins/AviaryScheddPlugin-plugin.so
  11/10/11 13:54:54 (pid:31234) axis2_ssl_utils_initialize_ctx failed
  11/10/11 13:54:54 (pid:31234) SSL/TLS requested but configuration failed
  11/10/11 13:54:54 (pid:31234) ERROR "Unable to configure AviaryProvider. Exiting..." at line 57 in file /builddir/build/BUILD/condor-7.6.4/src/condor_contrib/aviary/src/AviaryScheddPlugin.cpp
  Stack dump for process 31234 at timestamp 1320929694 (11 frames)
  condor_schedd(dprintf_dump_stack+0x44)[0x81a74c4]
  condor_schedd[0x8192537]
  [0xb89400]
  /lib/libc.so.6(abort+0xdd)[0xc023ad]
  condor_schedd(_EXCEPT_+0xb2)[0x8195832]
  /usr/lib/condor/plugins/AviaryScheddPlugin-plugin.so(_ZN6aviary3job18AviaryScheddPlugin15earlyInitializeEv+0x22f)[0x14c95f]
  condor_schedd(_ZN19ScheddPluginManager15EarlyInitializeEv+0x4c)[0x80e41fc]
  condor_schedd(_Z9main_initiPPc+0x75)[0x8111755]
  condor_schedd(main+0x13b5)[0x8131245]
  /lib/libc.so.6(__libc_start_main+0xe6)[0xbecce6]
  condor_schedd[0x80b5981]

Exception about SSL config problem is generated.

Output on RHEL5 i386, RHEL5 x86_64 and RHEL6 x86_64 is similar.


>>> VERIFIED