Bug 1234487

It looks to me as it's the server side initiated renegotiation that is broken.

Simple reproducer:
In console one:
openssl req -x509 -newkey rsa:2048 -keyout localhost.key -out localhost.crt -subj /CN=localhost -nodes -batch -sha256
openssl s_server -key localhost.key -cert localhost.crt

In console two:
openssl s_client

after connection is established, do a renegotiation by entering "R<enter>", the result should be:

depth=0 CN = localhost
verify error:num=18:self signed certificate
verify return:1
depth=0 CN = localhost
verify return:1

Now go back to console one, enter there "R<enter>", the connection will be aborted with following message:
SSL_do_handshake -> 1
ERROR
140398384944968:error:140943F2:SSL routines:SSL3_READ_BYTES:sslv3 alert unexpected message:s3_pkt.c:1259:SSL alert number 10
140398384944968:error:140940E5:SSL routines:SSL3_READ_BYTES:ssl handshake failure:s3_pkt.c:1346:
shutting down SSL

The test case that Hubert shows may be exhibiting an independent issue.  I've now traced through the behavior seen in Postgres, and confirmed that we are successfully executing multiple renegotiation cycles (started by calling SSL_renegotiate(), and ended by observing that SSL_num_renegotiations() has advanced).  The number of such cycles does not seem to matter.  What seems to be happening is that once the server has transmitted 2GB of data, if there have been renegotiations, then the next SSL_read() call fails with SSLerrmessage() yielding "sslv3 alert unexpected message".

Also, some further information: Postgres 9.4 and later behave differently than previous branches.  In 9.3.x and earlier the code that calls SSL_renegotiate() is structured quite a lot differently, and it seems to fail at the second renegotiation attempt.  So the external symptom looks nearly the same, but it can be triggered with just a little bit of data if you make ssl_renegotiation_limit small but not zero.  What I described above is the behavior in 9.4 and later.  I do not know whether the openssl bug you've acknowledged finding explains both symptoms.

I believe the manifestation is different for different Postgres branches but the underlying problem is the same. The renegotiation failure might depend on whether there are data buffered when the renegotiation is triggered. There can be probably cases where the renegotiation succeeds even with the broken openssl package - it depends on ordering and/or packing of the messages sent by the TLS server to the client.

I'll try testing with newer Postgres to be completely sure though.

(In reply to Tom Lane from comment #9)
> The test case that Hubert shows may be exhibiting an independent issue. 
> I've now traced through the behavior seen in Postgres, and confirmed that we
> are successfully executing multiple renegotiation cycles (started by calling
> SSL_renegotiate(),

It's always that the client sends the Client Hello request to the server to renegotiate.

But for server initiated renegotiation, server sends a hello_request first. This message is what causes the failure on client side, as a response the client sends an unexpected message alert to the server and closes the connection. The server will then report _alert_ unexpected message to the application.

In other words, there are two different error messages: "alert unexpected message" and "unexpected message". Please check if you see the former at the server and the latter at the client.

(In reply to Tomas Mraz from comment #11)
> I believe the manifestation is different for different Postgres branches but
> the underlying problem is the same. The renegotiation failure might depend
> on whether there are data buffered when the renegotiation is triggered.
> There can be probably cases where the renegotiation succeeds even with the
> broken openssl package - it depends on ordering and/or packing of the
> messages sent by the TLS server to the client.
> 
> I'll try testing with newer Postgres to be completely sure though.

There is a bug in OpenSSL where the application data and handshakes can't be intertwined, and I don't believe it is fixed: https://rt.openssl.org/Ticket/Display.html?id=3712&user=guest&pass=guest

but it did exist for a long time in OpenSSL, so the package update shouldn't trigger postgresql failures

I've tested the fixed packages with postgreSQL-9.4 and the pg_dump completes fine there.

(In reply to Hubert Kario from comment #12)
> In other words, there are two different error messages: "alert unexpected
> message" and "unexpected message". Please check if you see the former at the
> server and the latter at the client.

See the log snippets in the initial report.  The text after "SSL error: " is verbatim from SSLerrmessage() on each end.  The server-side report is not quite what you say here.

> There is a bug in OpenSSL where the application data and handshakes can't be
> intertwined, and I don't believe it is fixed:
> https://rt.openssl.org/Ticket/Display.html?id=3712&user=guest&pass=guest
> 
> but it did exist for a long time in OpenSSL, so the package update shouldn't
> trigger postgresql failures

Agreed, that one is unlikely to be our immediate problem.

(In reply to Tom Lane from comment #16)
> (In reply to Hubert Kario from comment #12)
> > In other words, there are two different error messages: "alert unexpected
> > message" and "unexpected message". Please check if you see the former at the
> > server and the latter at the client.
> 
> See the log snippets in the initial report.  The text after "SSL error: " is
> verbatim from SSLerrmessage() on each end.  The server-side report is not
> quite what you say here.

but it matches exactly what I provided in comment #5: "sslv3 alert unexpected message", the important distinction is presence or absence of "alert" in the error message.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-1398.html