The same scenario below is applicable to GnuTLS packages in Fedora 25 (gnutls-3.5.10-1.fc25), 26 (gnutls-3.5.10-1.fc26) and rawhide (gnutls-3.5.10-1.fc27).

+++ This bug was initially created as a clone of Bug #1434091 +++

Description of problem:
When OpenSSL acts as a server and client certificates are used, session renegotiation ends with an unexpected error.

Version-Release number of selected component (if applicable):
openssl-1.0.2k-4.el7.x86_64

How reproducible:
always

Steps to Reproduce:
See attached reproducer.

Actual results:
## Full log can be found in the attachments
# openssl s_server -www -key server.key -cert server.pem -CAfile ca.pem -Verify 1 -verify_return_error >server.log 2>server.err &
# sleep 2
# gnutls-cli --rehandshake --x509cafile ca.pem --x509keyfile client.key --x509certfile client.pem --port 4433 -d 500 localhost < /dev/null
<...TRUNCATED...>
|<4>| HSK[0x1f93210]: FINISHED (20) was received. Length 12[12], frag offset 0, frag length: 12, sequence: 0
|<5>| REC[0x1f93210]: Start of epoch cleanup
|<5>| REC[0x1f93210]: Epoch #0 freed
|<5>| REC[0x1f93210]: End of epoch cleanup
- Description: (TLS1.2)-(ECDHE-RSA-SECP256R1)-(AES-128-GCM)
- Session ID: 3C:D3:86:B4:28:FD:4A:46:F0:73:A1:87:9C:53:BF:CA:A9:96:D9:26:58:31:A2:B8:97:11:9B:35:36:31:5E:2E
|<3>| ASSERT: server_name.c:292
- Ephemeral EC Diffie-Hellman parameters
 - Using curve: SECP256R1
 - Curve size: 256 bits
- Version: TLS1.2
- Key Exchange: ECDHE-RSA
- Server Signature: RSA-SHA256
- Client Signature: RSA-SHA512
- Cipher: AES-128-GCM
- MAC: AEAD
- Compression: NULL
|<3>| ASSERT: status_request.c:382
- Options: safe renegotiation,
|<3>| ASSERT: srtp.c:317
|<3>| ASSERT: alpn.c:222
- Handshake was completed

- Simple Client Mode:

|<3>| ASSERT: gnutls_constate.c:586
|<5>| REC[0x1f93210]: Allocating epoch #2
<..TRUNCATED...>
|<5>| REC[0x1f93210]: Preparing Packet ChangeCipherSpec(20) with length: 1 and min pad: 0
|<9>| ENC[0x1f93210]: cipher: AES-128-GCM, MAC: AEAD, Epoch: 1
|<11>| WRITE: enqueued 30 bytes for 0x4. Total 1123 bytes.
|<5>| REC[0x1f93210]: Sent Packet[6] ChangeCipherSpec(20) in epoch 1 and length: 30
|<11>| HWRITE: wrote 1 bytes, 16 bytes left.
|<5>| REC[0x1f93210]: Preparing Packet Handshake(22) with length: 16 and min pad: 0
|<9>| ENC[0x1f93210]: cipher: AES-128-GCM, MAC: AEAD, Epoch: 2
|<11>| WRITE: enqueued 45 bytes for 0x4. Total 1168 bytes.
|<5>| REC[0x1f93210]: Sent Packet[1] Handshake(22) in epoch 2 and length: 45
|<11>| HWRITE: wrote 1 bytes, 0 bytes left.
|<11>| WRITE FLUSH: 1168 bytes in buffer.
|<11>| WRITE: wrote 1168 bytes, 0 bytes left.
|<3>| ASSERT: gnutls_buffers.c:1138
|<10>| READ: Got 5 bytes from 0x4
|<10>| READ: read 5 bytes from 0x4
|<10>| RB: Have 0 bytes into buffer. Adding 5 bytes.
|<10>| RB: Requested 5 bytes
|<5>| REC[0x1f93210]: SSL 3.3 Alert packet received. Epoch 0, length: 26
|<5>| REC[0x1f93210]: Expected Packet Handshake(22)
|<5>| REC[0x1f93210]: Received Packet Alert(21) with length: 26
|<10>| READ: Got 26 bytes from 0x4
|<10>| READ: read 26 bytes from 0x4
|<10>| RB: Have 5 bytes into buffer. Adding 26 bytes.
|<10>| RB: Requested 31 bytes
|<5>| REC[0x1f93210]: Decrypted Packet[5] Alert(21) with length: 2
|<5>| REC[0x1f93210]: Alert[2|10] - Unexpected message - was received
|<3>| ASSERT: gnutls_record.c:800
|<3>| ASSERT: gnutls_record.c:807
|<3>| ASSERT: gnutls_record.c:1327
|<3>| ASSERT: gnutls_buffers.c:1392
|<3>| ASSERT: gnutls_handshake.c:1428
|<3>| ASSERT: session_ticket.c:663
|<3>| ASSERT: gnutls_handshake.c:2846
*** Fatal error: A TLS fatal alert has been received.
*** Received alert [10]: Unexpected message
|<5>| REC: Sending Alert[2|80] - Internal error
<...TRUNCATED...>

# cat server.err
verify depth is 1, must return a certificate
depth=1 CN = CA
verify return:1
depth=0 CN = client
verify return:1
139873875388320:error:1408E0F4:SSL routines:ssl3_get_message:unexpected message:s3_both.c:408:

Expected results:
Renegotiation should not fail (which currently happens only when OpenSSL acts as a client).

--- Additional comment from Nikos Mavrogiannopoulos on 2017-03-21 09:27:23 CET ---

Adding the -state to the server, it gives the following output:

verify depth is 1, must return a certificate
SSL_accept:before/accept initialization
SSL_accept:SSLv3 read client hello A
SSL_accept:SSLv3 write server hello A
SSL_accept:SSLv3 write certificate A
SSL_accept:SSLv3 write key exchange A
SSL_accept:SSLv3 write certificate request A
SSL_accept:SSLv3 flush data
depth=1 CN = CA
verify return:1
depth=0 CN = client
verify return:1
SSL_accept:SSLv3 read client certificate A
SSL_accept:SSLv3 read client key exchange A
SSL_accept:SSLv3 read certificate verify A
SSL_accept:SSLv3 read finished A
SSL_accept:SSLv3 write session ticket A
SSL_accept:SSLv3 write change cipher spec A
SSL_accept:SSLv3 write finished A
SSL_accept:SSLv3 flush data
SSL_accept:before accept initialization
SSL_accept:SSLv3 read client hello A
SSL_accept:SSLv3 write server hello A
SSL_accept:SSLv3 write certificate A
SSL_accept:SSLv3 write key exchange A
SSL_accept:SSLv3 write server done A
SSL_accept:SSLv3 flush data
SSL_accept:SSLv3 read client certificate A
SSL3 alert write:fatal:unexpected_message
SSL_accept:error in SSLv3 read client key exchange A
139897927280504:error:1408E0F4:SSL routines:ssl3_get_message:unexpected message:s3_both.c:408:

So it seems that gnutls sends a client certificate in a renegotiation, even if it was not requested (no certificate request was sent). I guess that's an issue on gnutls rather than openssl.

--- Additional comment from Frantisek Sumsal on 2017-03-21 11:37:53 CET ---

Thank you, Nikos, I wasn't sure which side is to blame here as I couldn't reproduce this with NSS server - that was actually caused by a typo on my side. Regarding the new results, the issue indeed seems to be in GnuTLS:

## Server and client settings
# /usr/lib64/nss/unsupported-tools/selfserv -v -rr -d sql:./nssdb -p 4433 -V tls1.0: -S server
# gnutls-cli --rehandshake --x509cafile ca.pem --x509keyfile client.key --x509certfile client.pem --port 4433 -d 500 localhost

## NSS logs
selfserv: About to call accept.

selfserv: About to call accept.
selfserv: -- SSL3: Certificate Invalid, err -8172.
Peer's certificate issuer has been marked as not trusted by the user.
selfserv: HDX PR_Read returned error -8172:
Peer's certificate issuer has been marked as not trusted by the user.

However, when selfserv is started with -rr and -rrrr arguments (request and require a client certificate on initial (-rr) and second (-rrrr) handshake) the handshake is completed successfully.