Created attachment 121152 [details] test program tarball
Description of problem: If a server and client both use : ssl_ctx = SSL_CTX_new( SSLv23_method() ) to initialize the ssl context, and both set the SSL_OP_NO_SSLv2 ssl option : sslopt = SSL_OP_ALL | SSL_OP_NO_SSLv2 ; SSL_CTX_set_options(ssl, sslopt); then the client cannot connect to the server. Eg. for a server that can to support both v2 and v3 clients, but can be configured to support only v3 clients, as for the tog-pegasus cimserver, for which the FC-5 upgrade to openssl-0.9.8a broke SSL support . The example program in fc5_ssl_bug.tar.gz demonstrates the problem: $ gunzip < fc5_ssl_bug.tar.gz | tar -xpf - $ cd ssl $ make ... $ B='' make test ./svr & echo $!>svr.pid ;sleep 1; ./cli ; SSL_accept failure: -1 1 7982:error:1408F455:lib(20):func(143):reason(1109):s3_pkt.c:426: SSL_accept failed ... $ make test ./svr -b & echo $!>svr.pid ;sleep 1; ./cli -b; --- CLIENT 127.0.0.1 CONNECTED hello world --- CLIENT 127.0.0.1 DISCONNECTED kill `cat svr.pid`; rm -f svr.pid Without supplying the -b option test programs, they do: ssl_ctx = SSL_CTX_new( SSLv3_method() ); ... sslopt = SSL_OP_ALL ; SSL_CTX_set_options(ssl, sslopt); This makes them work OK . With the -b option, they do: ssl_ctx = SSL_CTX_new( SSLv23_method() ); ... sslopt = SSL_OP_ALL | SSL_OP_NO_SSLv2 ; SSL_CTX_set_options(ssl, sslopt); and this makes the server fails with the SSL_accept failure , whereas under 0.9.7f, or on FC-4, the above options work fine. Version-Release number of selected component (if applicable): openssl-0.9.8a How reproducible: 100% Steps to Reproduce: Run test program in attached 'fc5_ssl_bug.tar.gz' : $ gunzip < fc5_ssl_bug.tar.gz | tar -xpf - $ cd ssl $ make $ B='' make test Actual results: Server fails with SSL_accept error Expected results: Same results as for $ make test Additional info: See attached fc5_ssl_bug.tar.gz
Apparently the problem happens now because zlib compression is enabled by default in OpenSSL 0.9.8. (If its explicitely enabled with the OpenSSL 0.9.7 the problem happens there too.) I've switched it off for now. Please try openssl-0.9.8a-3 if it resolves the problem.
Yes, openssl-0.9.8a-3 resolves the problem - thanks!