Bug 1300773

Summary:

RFE: add support for native TLS encryption on chardev TCP transports

Product:

Red Hat Enterprise Linux 7

Reporter:

Daniel Berrangé <berrange>

Component:

qemu-kvm-rhev

Assignee:

Daniel Berrangé <berrange>

Status:

CLOSED ERRATA

QA Contact:

Min Deng <mdeng>

Severity:

unspecified

Docs Contact:

Priority:

unspecified

Version:

7.3

CC:

amit.shah, berrange, chayang, hhuang, huding, jbelka, jen, juzhang, knoel, lmiksik, mrezanin, pezhang, qizhu, qzhang, sherold, virt-maint, xfu

Target Milestone:

Keywords:

FutureFeature

Target Release:

---

Hardware:

Unspecified

OS:

Unspecified

Whiteboard:

Fixed In Version:

qemu-kvm-rhev-2.6.0-27.el7

Doc Type:

Enhancement

Doc Text:

Story Points:

---

Clone Of:

Clones:

1300776 (view as bug list)

Environment:

Last Closed:

2016-11-07 20:56:30 UTC

Type:

Bug

Regression:

---

Mount Type:

---

Documentation:

---

CRM:

Verified Versions:

Category:

---

oVirt Team:

---

RHEL 7.3 requirements from Atomic Host:

Cloudforms Team:

---

Target Upstream Version:

Embargoed:

Bug Depends On:

Bug Blocks:

1300776

Attachments:

Description	Flags
Create gnutls-server	none
createserver	none

Description Daniel Berrangé 2016-01-21 16:56:52 UTC

Description of problem:
The character device TCP backend currently transfers all data in plain text, which offers no security protection, unless it is tunnelled over an external service like SSH. Such tunnelling is inefficient and adds management complexity. Thus it is desired to have native support for TLS encryption with the TCP backend for character devices.

Current patches for this are:

  https://lists.gnu.org/archive/html/qemu-devel/2016-01/msg03396.html

Comment 2 Amit Shah 2016-06-06 10:04:20 UTC

This was added in qemu 2.6.

I suspect this bug is a dup of bug 1154115.

In any case, I'll just flip this to MODIFIED.

Comment 5 Min Deng 2016-08-23 08:33:23 UTC

Hi Developer,
   Could you please give some use case to verify this bug ? Thanks a lot.
Best Regards
Min

Comment 6 Daniel Berrangé 2016-08-23 13:42:29 UTC

This blog post describes the basic usage  https://www.berrange.com/posts/2016/08/16/improving-qemu-security-part-6-tls-support-for-character-devices/

I'll work on writing a c comment with more detailed testing suggestions too

Comment 7 Amit Shah 2016-08-24 19:19:32 UTC

Hope comment 6 is helpful.  If you need more info, feel free to ask.

Comment 8 Daniel Berrangé 2016-08-24 21:29:15 UTC

So in terms of test plans here are the key things to consider

 1. Run a generic gnutls-serv TLS server. Launch QEMU with a serial port as a TLS client, and ensure BIOS boot messages appear in the shell running gnutls-serv

 2. Launch QEMU with a serial port as a TLS server. Run a generic gnutls-cli TLS client, and ensure BIOS boot messages appear in the gnutls-cli terminal

Neither of these require a full OS

 3. Launch QEMU with a RHEL guest, and a serial port as a TLS server. Inside the guest OS run "tail -f /dev/ttyS0'. Then launch a second QEMU instance with a serial port as a TLS client, connecting to the first QEMU guest. The BIOS boot messages from the second QEMU should appear in the guest terminal where you're running 'tail -f /dev/ttyS0'.

These prove general correct operation of the chardevs with serial port TLS setup.

You also want some negative tests, using the setup from item 3

 4. As 3, but have the TLS server certificate that only contains "localhost". Then have the second QEMU connect to "127.0.0.1", it should fail, since 127.0.0.1 doesn't match "localhost"

 5. Try to connect 2 QEMU instances with one chardev in TCP clint mode and the other in server mode, but with the TLS credentials for *both* QEMUs using endpoint=client. They should fail to complete the handshake, since you must normally have different "endpoint" values.

Comment 9 Amit Shah 2016-09-07 07:21:51 UTC

*** Bug 1154115 has been marked as a duplicate of this bug. ***

Comment 10 Min Deng 2016-09-13 08:26:19 UTC

According to above scenarios QE did the following tests,
scenario 1,
1.start a gnutls-server
#gnutls-serv --http             --x509cafile x509-ca.pem             --x509keyfile x509-server-key.pem             --x509certfile x509-server.pem
2.create related files under by certool command
#ls
ca-cert.pem    ca.tmpl      proxy.tmpl   x509-ca-key.pem  x509-client-key.pem  x509-client.pem     x509-proxy.pem       x509-server.pem
ca-client.pem  client.tmpl  server.tmpl  x509-ca.pem      x509-client.p12      x509-proxy-key.pem  x509-server-key.pem
3.ln -s x509-ca.pem ca-cert.pem (create a symbol for or do copy)
 #lrwxrwxrwx. 1 root root   11 Sep 13 02:24 ca-cert.pem -> x509-ca.pem  - 
4.boot up guest with 
/usr/libexec/qemu-kvm -object tls-creds-x509,id=tls0,endpoint=client,dir=/home/TLS-server -chardev socket,id=s0,host=127.0.0.1/localhost,port=5556,tls-creds=tls0 -device isa-serial,chardev=s0
Expected results,
BIOS boot messages appear in the shell running gnutls-serv
Actual results,
...
* Accepted connection from IPv4 127.0.0.1 port 33426 on Tue Sep 13 02:53:29 2016
Error in handshake
Error: The TLS connection was non-properly terminated.

* Accepted connection from IPv4 127.0.0.1 port 33428 on Tue Sep 13 02:53:29 2016
Error in handshake
Error: The TLS connection was non-properly terminated.
...

scenario 2,
1.boot up guest with the following cli,(make symbols or do copy for them)
  /usr/libexec/qemu-kvm -object tls-creds-x509,id=tls0,endpoint=server,dir=/home/TLS-server -chardev socket,id=s0,host=localhost,port=12345,tls-creds=tls0,server -device isa-serial,chardev=s0
2.connect TLS-server via 
  gnutls-cli --priority=NORMAL -p 12345 --x509cafile=/home/TLS-server/ca-cert.pem localhost

Expected results,
 The server is connected
Actual results,
Timeout occurred
Processed 1 CA certificate(s).
Resolving 'localhost'...
Connecting to '::1:12345'...
*** Fatal error: The operation timed out
*** Handshake has failed
GnuTLS error: The operation timed out

scenario 3,scenario 4 and scenario 5
To my opinions,above results would affect these.

Comment 11 Min Deng 2016-09-13 08:29:19 UTC

Created attachment 1200386 [details]
Create gnutls-server

Create gnutls-server

Comment 12 Daniel Berrangé 2016-09-14 09:14:18 UTC

In the first scenario your script is creating the server cert with a hostname of "test.gnutls.org" and you're connecting to localhost and/or 127.0.0.1. As such the certificate verification is failing and QEMU is dropping the connection. In the second scenario it looks like the TCP connection isn't even succeeding, possibly because youve used the wrong filenames for certs - it must be server-cert.pem not x509-server.pem

Comment 13 Min Deng 2016-09-18 00:10:27 UTC

(In reply to Daniel Berrange from comment #12)
> In the first scenario your script is creating the server cert with a
> hostname of "test.gnutls.org" and you're connecting to localhost and/or
> 127.0.0.1. As such the certificate verification is failing and QEMU is
> dropping the connection. In the second scenario it looks like the TCP
> connection isn't even succeeding, possibly because youve used the wrong
> filenames for certs - it must be server-cert.pem not x509-server.pem
  For this part,I do really renamed it by server-cert in my steps

Comment 14 Min Deng 2016-09-18 00:26:01 UTC

(In reply to dengmin from comment #13)
> (In reply to Daniel Berrange from comment #12)
> > In the first scenario your script is creating the server cert with a
> > hostname of "test.gnutls.org" and you're connecting to localhost and/or
> > 127.0.0.1. As such the certificate verification is failing and QEMU is
> > dropping the connection. 
   Actually,there is another field for set hostname instead but not this one.I just let it as default.

Comment 15 Daniel Berrangé 2016-09-20 09:07:12 UTC

(In reply to dengmin from comment #14)
> (In reply to dengmin from comment #13)
> > (In reply to Daniel Berrange from comment #12)
> > > In the first scenario your script is creating the server cert with a
> > > hostname of "test.gnutls.org" and you're connecting to localhost and/or
> > > 127.0.0.1. As such the certificate verification is failing and QEMU is
> > > dropping the connection. 
>    Actually,there is another field for set hostname instead but not this
> one.I just let it as default.

Your script from comment #11 is clearly wrong - it is only setting the CN field, to the wrong value.

Comment 16 Min Deng 2016-09-21 09:05:53 UTC

Hi Daniel,
   Thanks for your help.According to what we talked about in the bug I did a summary of those steps in these days.After setting such a gnutl server I still got the same results to comment10.I guess you should also have your own steps for setting up cert,right ?if so could you please share it with me ? Many thank !
   I've tried it on more than two hosts so far,the build I used was qemu-kvm-rhev-2.6.0-26.el7.x86_64
I.The next step is to add support for X.509. First we generate a CA:
 1.certtool --generate-privkey > x509-ca-key.pem
 2.echo 'cn = hp-dl385pg8-08.lab.eng.pek2.redhat.com' > ca.tmpl
 3.echo 'ca' >> ca.tmpl
 4.echo 'cert_signing_key' >> ca.tmpl
 5.certtool --generate-self-signed --load-privkey x509-ca-key.pem --template ca.tmpl --outfile x509-ca.pem

II.Then generate a server certificate. Remember to change the dns_name value to the name of your server host, or skip that command to avoid the field.
1.certtool --generate-privkey > x509-server-key.pem
2.echo 'organization = GnuTLS test server' > server.tmpl
3.echo 'cn = hp-dl385pg8-08.lab.eng.pek2.redhat.com' >> server.tmpl
4.echo 'tls_www_server' >> server.tmpl
5.echo 'encryption_key' >> server.tmpl
6.echo 'signing_key' >> server.tmpl
7.echo 'dns_name = hp-dl385pg8-08.lab.eng.pek2.redhat.com' >> server.tmpl
8.certtool --generate-certificate --load-privkey x509-server-key.pem --load-ca-certificate x509-ca.pem --load-ca-privkey x509-ca-key.pem --template server.tmpl --outfile x509-server.pem

III.For use in the client, you may want to generate a client certificate as well.
1.certtool --generate-privkey > x509-client-key.pem
2.echo 'cn = hp-dl385pg8-08.lab.eng.pek2.redhat.com' > client.tmpl
3.echo 'tls_www_client' >> client.tmpl
4.echo 'encryption_key' >> client.tmpl
5.echo 'signing_key' >> client.tmpl
6.certtool --generate-certificate --load-privkey x509-client-key.pem --load-ca-certificate x509-ca.pem --load-ca-privkey x509-ca-key.pem --template client.tmpl --outfile x509-client.pem

IIII.To be able to import the client key/certificate into some applications, you will need to convert them into a PKCS#12 structure. This also encrypts the security sensitive key with a password.
1.certtool --to-p12 --load-ca-certificate x509-ca.pem --load-privkey x509-client-key.pem --load-certificate x509-client.pem --outder --outfile x509-client.p12

here - key name is redhat
       password redhat

IV.
gnutls-serv --http \
            --x509cafile x509-ca.pem \
            --x509keyfile x509-server-key.pem \
            --x509certfile x509-server.pem

Could you please have a look on these mapping relationship ? Thanks again.
ca-cert.pem -> x509-ca.pem
server-cert.pem - > x509-server.pem
server-key.pem -> x509-server-key.pem

Run server:
gnutls-serv --http \
            --x509cafile ca-cert.pem \
            --x509keyfile server-key.pem \
            --x509certfile server-cert.pem

Comment 17 Daniel Berrangé 2016-09-21 10:00:02 UTC

It appears we backported a patch which has broken TLS chardevs :-(

kvm-char-add-and-use-tcp_chr_wait_connected.patch


commit d7a04fd7d500870ad70f15089c6590e4bea73e70
Author: Marc-André Lureau <marcandre.lureau>
Date:   Wed Jul 27 01:15:18 2016 +0400

    char: add and use tcp_chr_wait_connected
    
    Add a chr_wait_connected for the tcp backend, and use it in the
    open_socket() function.
    
    Signed-off-by: Marc-André Lureau <marcandre.lureau>
    Reviewed-by: Michael S. Tsirkin <mst>
    Signed-off-by: Michael S. Tsirkin <mst>


We need to fix that with

commit 1dc8a6695c731abb7461c637b2512c3670d82be4
Author: Marc-André Lureau <marcandre.lureau>
Date:   Tue Aug 16 12:33:32 2016 +0400

    char: fix waiting for TLS and telnet connection
    
    Since commit d7a04fd7d5008, tcp_chr_wait_connected() was introduced,
    so vhost-user could wait until a backend started successfully. In
    vhost-user case, the chr socket must be plain unix, and the chr+vhost
    setup happens synchronously during qemu startup.
    
    However, with TLS and telnet socket, initial socket setup happens
    asynchronously, and s->connected is not set after the socket is
    accepted. In order for tcp_chr_wait_connected() to not keep accepting
    new connections and proceed with the last accepted socket, it can
    check for s->ioc instead.
    
    Signed-off-by: Marc-André Lureau <marcandre.lureau>
    Reviewed-by: Daniel P. Berrange <berrange>
    Message-id: 20160816083332.15088-1-marcandre.lureau
    Signed-off-by: Peter Maydell <peter.maydell>

Comment 19 Miroslav Rezanina 2016-09-23 13:58:24 UTC

Fix included in qemu-kvm-rhev-2.6.0-27.el7

Comment 20 Min Deng 2016-09-26 06:33:28 UTC

Hi Daniel,
   Sorry for delay since 24,25 are our weekend days.I did some tests about this bug today and QE want to confirm them with you to check if all these tests are enough or not,unfortunately some failure still exist so far,could you please help to check ? Any issues please let me know!
*Scenario 1,
1.set up a gnutl server and detail info please refer to comment16.
  gnutls-serv --http \
            --x509cafile ca-cert.pem \
            --x509keyfile server-key.pem \
            --x509certfile server-cert.pem 
2.boot up guest with cli
  /usr/libexec/qemu-kvm -object tls-creds-x509,id=tls0,endpoint=client,dir=/home/mdeng -chardev socket,id=s0,host=hp-dl385pg8-08.lab.eng.pek2.redhat.com,port=5556,tls-creds=tls0 -device isa-serial,chardev=s0

Actual results,
Phenomenon from qemu-kvm was following as below,
VNC server running on '::1;5900'
Phenomenon from server was following as below,
* Accepted connection from IPv4 10.73.196.143 port 48226 on Sun Sep 25 23:07:08 2016
- Description: (TLS1.2)-(ECDHE-RSA-SECP256R1)-(AES-128-GCM)
- Session ID: DD:69:EE:27:CB:CC:06:5A:4E:C5:53:AC:BD:AC:16:74:72:DF:57:0B:EA:1A:6A:08:29:BA:B1:2F:73:3B:8D:04
No certificates found!
- Ephemeral EC Diffie-Hellman parameters
 - Using curve: SECP256R1
 - Curve size: 256 bits
- Version: TLS1.2
- Key Exchange: ECDHE-RSA
- Server Signature: RSA-SHA256
- Cipher: AES-128-GCM
- MAC: AEAD
- Compression: NULL
- Options: safe renegotiation,
- Channel binding 'tls-unique': c2f498371960bbe4127730b7
Scheduling inactive connection for close

Expected results,the qemu-kvm connects to server successfully.

*Scenario 2,
1.Launch QEMU with a serial port as a TLS server. 
  boot up guest with cli
  /usr/libexec/qemu-kvm -object tls-creds-x509,id=tls0,endpoint=server,dir=/home/mdeng -chardev socket,id=s0,host=hp-dl385pg8-08.lab.eng.pek2.redhat.com,port=13131,tls-creds=tls0,server -device isa-serial,chardev=s0
2.run gntuls client to connect it.
  gnutls-cli --priority=NORMAL -p 13131 --x509cafile=/home/mdeng/ca-cert.pem hp-dl385pg8-08.lab.eng.pek2.redhat.com
Actual results,
[root@hp-dl385pg8-08 mdeng]# gnutls-cli --priority=NORMAL -p 13131 --x509cafile=/home/mdeng/ca-cert.pem hp-dl385pg8-08.lab.eng.pek2.redhat.com
Processed 1 CA certificate(s).
Resolving 'hp-dl385pg8-08.lab.eng.pek2.redhat.com'...
Connecting to '10.73.196.143:13131'...
- Certificate type: X.509
- Got a certificate list of 1 certificates.
- Certificate[0] info:
 - subject `CN=hp-dl385pg8-08.lab.eng.pek2.redhat.com', issuer `CN=hp-dl385pg8-08.lab.eng.pek2.redhat.com', RSA key 2048 bits, signed using RSA-SHA256, activated `2016-09-26 02:55:25 UTC', expires `2017-09-26 02:55:25 UTC', SHA-1 fingerprint `0345848d39214a3e4675a807c669b99f32f0049b'
	Public Key ID:
		f871b3b531555aa4e4d21d77265abc4e9887a8ad
	Public key's random art:
		+--[ RSA 2048]----+
		|             .+oB|
		|             *.O+|
		|           .o+*..|
		|       .  . +o+  |
		|      . Soo ++   |
		|       ..o.+ +.  |
		|        ... .    |
		|        E        |
		|                 |
		+-----------------+

- Status: The certificate is trusted. 
- Successfully sent 0 certificate(s) to server.
- Description: (TLS1.2)-(ECDHE-RSA-SECP256R1)-(AES-128-GCM)
- Session ID: 12:99:D0:9A:70:09:41:6E:3C:AB:DC:F9:20:C8:E8:36:AD:B3:27:80:1D:BB:C7:34:54:41:5A:AD:A8:F2:40:DE
- Ephemeral EC Diffie-Hellman parameters
 - Using curve: SECP256R1
 - Curve size: 256 bits
- Version: TLS1.2
- Key Exchange: ECDHE-RSA
- Server Signature: RSA-SHA256
- Cipher: AES-128-GCM
- MAC: AEAD
- Compression: NULL
- Options: safe renegotiation,
- Handshake was completed
- Simple Client Mode:
Expected results,Connect to gnutls created by qemu-kvm successfully

*Scenario 3 
1.boot up guest with cli
   /usr/libexec/qemu-kvm -m 8G -smp 16 -name vocado-vt-vm1 -vga std -device virtio-blk-pci,id=disk,drive=drive-disk0,bootindex=1 -drive file=RHEL-Server-7.3-64-virtio.raw,format=raw,if=none,id=drive-disk0,cache=none,werror=stop -vnc :1 -enable-kvm -monitor stdio -vnc :0 -object tls-creds-x509,id=tls0,endpoint=server,dir=/home/client -chardev socket,id=s0,host=hp-dl385pg8-08.lab.eng.pek2.redhat.com,port=12121,tls-creds=tls0,server,nowait -device isa-serial,chardev=s0
2.run tail -f /dev/ttyS0
3.boot up guest with cli
  /usr/libexec/qemu-kvm -m 8G -smp 16 -name vocado-vt-vm1 -vga std -device virtio-blk-pci,id=disk,drive=drive-disk0,bootindex=1 -drive file=RHEL-Server-7.3-64-virtio-2.raw,format=raw,if=none,id=drive-disk0,cache=none,werror=stop -vnc :2 -object tls-creds-x509,id=tls0,endpoint=client,dir=/home/mdeng -chardev socket,id=s0,host=hp-dl385pg8-08.lab.eng.pek2.redhat.com,port=12121,tls-creds=tls0 -device isa-serial,chardev=s0 -monitor stdio
  or
  /usr/libexec/qemu-kvm -object tls-creds-x509,id=tls0,endpoint=client,dir=/home/mdeng -chardev socket,id=s0,host=hp-dl385pg8-08.lab.eng.pek2.redhat.com,port=12121,tls-creds=tls0 -device isa-serial,chardev=s0 -monitor stdio

Expected results,there should be some boot message in the terminal where you were executing "tail -f /dev/ttyS0"
Actual results,there is not any information from terminal.

*Scenario 4
1.Let Certifications only contain "localhost"
2.connect to it by qemu-kvm instance 
   /usr/libexec/qemu-kvm -object tls-creds-x509,id=tls0,endpoint=client,dir=/home/mdeng -chardev socket,id=s0,host=127.0.0.1,port=5556,tls-creds=tls0 -device isa-serial,chardev=s0

Expected resulst,it fail,refer to comment 8
Actual results,
* Accepted connection from IPv4 127.0.0.1 port 35784 on Mon Sep 26 01:48:11 2016
- Description: (TLS1.2)-(ECDHE-RSA-SECP256R1)-(AES-128-GCM)
- Session ID: 84:BF:8D:10:5B:2B:91:39:31:E0:F7:54:C5:AB:F4:D3:3B:5E:32:A2:40:B0:E7:CD:EC:C7:51:68:C5:A7:8C:E8
No certificates found!
- Ephemeral EC Diffie-Hellman parameters
 - Using curve: SECP256R1
 - Curve size: 256 bits
- Version: TLS1.2
- Key Exchange: ECDHE-RSA
- Server Signature: RSA-SHA256
- Cipher: AES-128-GCM
- MAC: AEAD
- Compression: NULL
- Options: safe renegotiation,
- Channel binding 'tls-unique': b479d8e4a925b0baa2008c9c
Scheduling inactive connection for close

*Scenario 5
1.server mode,change it to client endpoint but qemu-kvm will be abandoned if set "endpoint = client" as soon as try to run qemu-kvm instance as server.
 /usr/libexec/qemu-kvm -object tls-creds-x509,id=tls0,endpoint=*client*,dir=/home/mdeng -chardev socket,id=s0,host=hp-dl385pg8-08.lab.eng.pek2.redhat.com,port=12121,tls-creds=tls0,server -device isa-serial,chardev=s0
qemu-kvm: -chardev socket,id=s0,host=hp-dl385pg8-08.lab.eng.pek2.redhat.com,port=13131,tls-creds=tls0,server: Expected TLS credentials for server endpoint
2.client mode,it cannot boot up because the qemu-kvm instance of server of step 1 was not available.Notes,According to comment 8 - >step 5,it need to be verified if it fail to do handshake or not when setting duplicate endpoint.QE don't think it possible to boot up such two qemu-kvm instances at the same time because of current behaviours of qemu-kvm  was not allowed.if I'm wrong please correct me.
  /usr/libexec/qemu-kvm -object tls-creds-x509,id=tls0,endpoint=client,dir=/home/mdeng -chardev socket,id=s0,host=hp-dl385pg8-08.lab.eng.pek2.redhat.com,port=5556,tls-creds=tls0 -device isa-serial,chardev=s0
qemu-kvm: -chardev socket,id=s0,host=hp-dl385pg8-08.lab.eng.pek2.redhat.com,port=5556,tls-creds=tls0: Failed to connect socket: Connection refused
3.client mode ,change it to server endpoint ,it is a negative test case as well.
 /usr/libexec/qemu-kvm -object tls-creds-x509,id=tls0,endpoint=server,dir=/home/mdeng -chardev socket,id=s0,host=hp-dl385pg8-08.lab.eng.pek2.redhat.com,port=5556,tls-creds=tls0 -device isa-serial,chardev=s0
qemu-kvm: -chardev socket,id=s0,host=hp-dl385pg8-08.lab.eng.pek2.redhat.com,port=5556,tls-creds=tls0: Expected TLS credentials for client endpoint

Best Regards,
Min

Comment 21 Min Deng 2016-09-26 10:00:27 UTC

Created attachment 1204758 [details]
createserver

Comment 22 Daniel Berrangé 2016-09-26 10:32:19 UTC

(In reply to dengmin from comment #20)
> Hi Daniel,
>    Sorry for delay since 24,25 are our weekend days.I did some tests about
> this bug today and QE want to confirm them with you to check if all these
> tests are enough or not,unfortunately some failure still exist so far,could
> you please help to check ? Any issues please let me know!
> *Scenario 1,
> 1.set up a gnutl server and detail info please refer to comment16.
>   gnutls-serv --http \
>             --x509cafile ca-cert.pem \
>             --x509keyfile server-key.pem \
>             --x509certfile server-cert.pem 

Using '--http' here means that the client is expected to run an HTTP handshake which QEMU certainly won't do. You should instead be using '--echo'

> 2.boot up guest with cli
>   /usr/libexec/qemu-kvm -object
> tls-creds-x509,id=tls0,endpoint=client,dir=/home/mdeng -chardev
> socket,id=s0,host=hp-dl385pg8-08.lab.eng.pek2.redhat.com,port=5556,tls-
> creds=tls0 -device isa-serial,chardev=s0

You've not told QEMU to put anything on the serial console, so gnutls-serv is not going to print anything regardless.

I run the following

$ cd /home/mdeng
$ gnutls-serv    --echo         --x509cafile ca-cert.pem             --x509keyfile server-key.pem             --x509certfile server-cert.pem 

and

$ /usr/libexec/qemu-kvm -object tls-creds-x509,id=tls0,endpoint=client,dir=/home/mdeng -chardev socket,id=s0,host=hp-dl385pg8-08.lab.eng.pek2.redhat.com,port=5556,tls-creds=tls0 -device isa-serial,chardev=s0 -display none -device sga


and gnutls-serv showed the QEMU BIOS boot messages as expected




> *Scenario 2,
> 1.Launch QEMU with a serial port as a TLS server. 
>   boot up guest with cli
>   /usr/libexec/qemu-kvm -object
> tls-creds-x509,id=tls0,endpoint=server,dir=/home/mdeng -chardev
> socket,id=s0,host=hp-dl385pg8-08.lab.eng.pek2.redhat.com,port=13131,tls-
> creds=tls0,server -device isa-serial,chardev=s0

This has not set 'verify-peer' so  by default QEMU will be expecting  the client to provide a certificate

> 2.run gntuls client to connect it.
>   gnutls-cli --priority=NORMAL -p 13131 --x509cafile=/home/mdeng/ca-cert.pem
> hp-dl385pg8-08.lab.eng.pek2.redhat.com

but you've not provided a certificate here.

Also, you've again not told QEMU to use the serial port for anything

I run qemu with:

 $ /usr/libexec/qemu-kvm -object tls-creds-x509,id=tls0,endpoint=server,dir=/home/mdeng -chardev socket,id=s0,host=hp-dl385pg8-08.lab.eng.pek2.redhat.com,port=9000,tls-creds=tls0,server -device isa-serial,chardev=s0 -device sga -display none

ANd then run

 $ gnutls-cli --priority=NORMAL -p9000 --x509cafile=/home/mdeng/ca-cert.pem hp-dl385pg8-08.lab.eng.pek2.redhat.com --x509certfile=/home/mdeng/x509-client.pem --x509keyfile=/home/mdeng/x509-client-key.pem

and it printed the QEMU BIOS boot messages as expected.



> *Scenario 3 
> 1.boot up guest with cli
>    /usr/libexec/qemu-kvm -m 8G -smp 16 -name vocado-vt-vm1 -vga std -device
> virtio-blk-pci,id=disk,drive=drive-disk0,bootindex=1 -drive
> file=RHEL-Server-7.3-64-virtio.raw,format=raw,if=none,id=drive-disk0,
> cache=none,werror=stop -vnc :1 -enable-kvm -monitor stdio -vnc :0 -object
> tls-creds-x509,id=tls0,endpoint=server,dir=/home/client -chardev
> socket,id=s0,host=hp-dl385pg8-08.lab.eng.pek2.redhat.com,port=12121,tls-
> creds=tls0,server,nowait -device isa-serial,chardev=s0


This does not have 'verify-peer' set so it'll be expecting the client to provide a certificate.


> 2.run tail -f /dev/ttyS0
> 3.boot up guest with cli
>   /usr/libexec/qemu-kvm -m 8G -smp 16 -name vocado-vt-vm1 -vga std -device
> virtio-blk-pci,id=disk,drive=drive-disk0,bootindex=1 -drive
> file=RHEL-Server-7.3-64-virtio-2.raw,format=raw,if=none,id=drive-disk0,
> cache=none,werror=stop -vnc :2 -object
> tls-creds-x509,id=tls0,endpoint=client,dir=/home/mdeng -chardev
> socket,id=s0,host=hp-dl385pg8-08.lab.eng.pek2.redhat.com,port=12121,tls-
> creds=tls0 -device isa-serial,chardev=s0 -monitor stdio
>   or
>   /usr/libexec/qemu-kvm -object
> tls-creds-x509,id=tls0,endpoint=client,dir=/home/mdeng -chardev
> socket,id=s0,host=hp-dl385pg8-08.lab.eng.pek2.redhat.com,port=12121,tls-
> creds=tls0 -device isa-serial,chardev=s0 -monitor stdio


The '/home/mdeng' directory does not contain the required client certificate files - eg no client-cert.pem or client-key.pem

so QEMU server will reject the connection



> *Scenario 4
> 1.Let Certifications only contain "localhost"
> 2.connect to it by qemu-kvm instance 
>    /usr/libexec/qemu-kvm -object
> tls-creds-x509,id=tls0,endpoint=client,dir=/home/mdeng -chardev
> socket,id=s0,host=127.0.0.1,port=5556,tls-creds=tls0 -device
> isa-serial,chardev=s0
> 
> Expected resulst,it fail,refer to comment 8
> Actual results,
> * Accepted connection from IPv4 127.0.0.1 port 35784 on Mon Sep 26 01:48:11
> 2016
> - Description: (TLS1.2)-(ECDHE-RSA-SECP256R1)-(AES-128-GCM)
> - Session ID:
> 84:BF:8D:10:5B:2B:91:39:31:E0:F7:54:C5:AB:F4:D3:3B:5E:32:A2:40:B0:E7:CD:EC:
> C7:51:68:C5:A7:8C:E8
> No certificates found!
> - Ephemeral EC Diffie-Hellman parameters
>  - Using curve: SECP256R1
>  - Curve size: 256 bits
> - Version: TLS1.2
> - Key Exchange: ECDHE-RSA
> - Server Signature: RSA-SHA256
> - Cipher: AES-128-GCM
> - MAC: AEAD
> - Compression: NULL
> - Options: safe renegotiation,
> - Channel binding 'tls-unique': b479d8e4a925b0baa2008c9c
> Scheduling inactive connection for close
> 
> *Scenario 5
> 1.server mode,change it to client endpoint but qemu-kvm will be abandoned if
> set "endpoint = client" as soon as try to run qemu-kvm instance as server.
>  /usr/libexec/qemu-kvm -object
> tls-creds-x509,id=tls0,endpoint=*client*,dir=/home/mdeng -chardev
> socket,id=s0,host=hp-dl385pg8-08.lab.eng.pek2.redhat.com,port=12121,tls-
> creds=tls0,server -device isa-serial,chardev=s0
> qemu-kvm: -chardev
> socket,id=s0,host=hp-dl385pg8-08.lab.eng.pek2.redhat.com,port=13131,tls-
> creds=tls0,server: Expected TLS credentials for server endpoint

That is expected. If you run QEMU as a TCP server, you must also give it TLS server endpoint certs


> 2.client mode,it cannot boot up because the qemu-kvm instance of server of
> step 1 was not available.Notes,According to comment 8 - >step 5,it need to
> be verified if it fail to do handshake or not when setting duplicate
> endpoint.QE don't think it possible to boot up such two qemu-kvm instances
> at the same time because of current behaviours of qemu-kvm  was not
> allowed.if I'm wrong please correct me.
>   /usr/libexec/qemu-kvm -object
> tls-creds-x509,id=tls0,endpoint=client,dir=/home/mdeng -chardev
> socket,id=s0,host=hp-dl385pg8-08.lab.eng.pek2.redhat.com,port=5556,tls-
> creds=tls0 -device isa-serial,chardev=s0
> qemu-kvm: -chardev
> socket,id=s0,host=hp-dl385pg8-08.lab.eng.pek2.redhat.com,port=5556,tls-
> creds=tls0: Failed to connect socket: Connection refused
> 3.client mode ,change it to server endpoint ,it is a negative test case as
> well.
>  /usr/libexec/qemu-kvm -object
> tls-creds-x509,id=tls0,endpoint=server,dir=/home/mdeng -chardev
> socket,id=s0,host=hp-dl385pg8-08.lab.eng.pek2.redhat.com,port=5556,tls-
> creds=tls0 -device isa-serial,chardev=s0
> qemu-kvm: -chardev
> socket,id=s0,host=hp-dl385pg8-08.lab.eng.pek2.redhat.com,port=5556,tls-
> creds=tls0: Expected TLS credentials for client endpoint

Again this is expected for same reason as above.

Comment 23 Min Deng 2016-09-26 16:35:31 UTC

Because Scenario 4/Scenario 5 work as expected QE re-test Scenario 1/Scenario 2/Scenario 3 
Scenario 1,
1.boot up gnutls server 
gnutls-serv    --echo         --x509cafile ca-cert.pem             --x509keyfile server-key.pem             --x509certfile server-cert.pem 
2.boot up qemu-kvm instance
/usr/libexec/qemu-kvm -object tls-creds-x509,id=tls0,endpoint=client,dir=/home/mdeng -chardev socket,id=s0,host=hp-dl385pg8-08.lab.eng.pek2.redhat.com,port=5556,tls-creds=tls0 -device isa-serial,chardev=s0 -display none -device sga
Expected results,
it printed the QEMU BIOS boot messages as expected.
Actual results,
* Accepted connection from IPv4 10.73.196.143 port 48300 on Mon Sep 26 10:39:15 2016
- Description: (TLS1.2)-(ECDHE-RSA-SECP256R1)-(AES-128-GCM)
- Session ID: 6A:A4:60:29:19:09:E8:7F:AC:85:22:7A:03:9F:37:AB:B4:59:A8:CC:CB:76:33:A7:FD:8F:17:A5:B9:53:3A:87
No certificates found!
- Ephemeral EC Diffie-Hellman parameters
 - Using curve: SECP256R1
 - Curve size: 256 bits
- Version: TLS1.2
- Key Exchange: ECDHE-RSA
- Server Signature: RSA-SHA256
- Cipher: AES-128-GCM
- MAC: AEAD
- Compression: NULL
- Options: safe renegotiation,
- Channel binding 'tls-unique': 6515376d2089e74aab0a0a48
received: 

*** Processing 2 bytes command: 

received: Google, Inc.

*** Processing 14 bytes command: Google, Inc.

received: Serial Graphics Adapter 12/29/13

*** Processing 34 bytes command: Serial Graphics Adapter 12/29/13

received: SGABIOS $Id: sgabios.S 8 2010-04-22 00:03:40Z nlaredo $ (mockbuild@) Sun Dec 29 03:43:06 UTC 2013

*** Processing 99 bytes command: SGABIOS $Id: sgabios.S 8 2010-04-22 00:03:40Z nlaredo $ (mockbuild@) Sun Dec 29 03:43:06 UTC 2013

received: 4 0

*** Processing 5 bytes command: 4 0

received: 
 eaBIOS (version 1.9.1-4.el7)

*** Processing 128 bytes command: 

 eaBIOS (version 1.9.1-4.el7)

received: 
iPXE (http://ipxe.org) 00:03.0 CA80 PCI2.10 PnP PMM+07F978B0+07EF78B0 CA80
                                                                               
received: sing 128 bytes command: 
received: sing 128 bytes command: 
Booting from Hard Disk...ommand: 
Boot failed: could not read the boot disk
received: sing 128 bytes command: 
Booting from Floppy...es command: 
Boot failed: could not read the boot disk
received: sing 128 bytes command: 
Booting from DVD/CD...es command: 
Boot failed: Could not read from CDROM (code 0003)
Booting from ROM...bytes command: 
iPXE (PCI 00:03.0) starting execution...ok
iPXE initialising devices...oknd: 
received: sing 128 bytes command: 
received: sing 128 bytes command: 
received: sing 17 bytes command: 
iPXE 1.0.0+ (6366fa7a) -- Open Source Network Boot Firmware -- http://ipxe.org
Features: DNS HTTP iSCSI TFTP AoE ELF MBOOT PXE bzImage Menu PXEXT
Features: DNS HTTP iSCSI TFTP AoE ELF MBOOT PXE bzImage Menu PXEXT
*** Processing 128 bytes command: 
received: 

*** Processing 17 bytes command: 

net0: 52:54:00:12:34:56 using 82540em on PCI00:03.0 (open)

net0: 52:54:00:12:34:56 using 82540em on PCI00:03.0 (open)

  [Link:up, TX:0 TXE:0 RX:0 RXE:0]

  [Link:up, TX:0 TXE:0 RX:0 RXE:0]

Scheduling inactive connection for close

Scenario 2,
1.boot up guest
  /usr/libexec/qemu-kvm -object tls-creds-x509,id=tls0,endpoint=server,dir=/home/mdeng -chardev socket,id=s0,host=hp-dl385pg8-08.lab.eng.pek2.redhat.com,port=9000,tls-creds=tls0,server -device isa-serial,chardev=s0 -device sga -display none
2.run 
 gnutls-cli --priority=NORMAL -p9000 --x509cafile=/home/mdeng/ca-cert.pem hp-dl385pg8-08.lab.eng.pek2.redhat.com --x509certfile=/home/mdeng/x509-client.pem --x509keyfile=/home/mdeng/x509-client-key.pem

Expected results,it printed the QEMU BIOS boot messages as expected
Actual results,Processed 1 CA certificate(s).
Processed 1 client X.509 certificates...
Resolving 'hp-dl385pg8-08.lab.eng.pek2.redhat.com'...
Connecting to '10.73.196.143:9000'...
- Certificate type: X.509
- Got a certificate list of 1 certificates.
- Certificate[0] info:
 - subject `CN=hp-dl385pg8-08.lab.eng.pek2.redhat.com', issuer `CN=hp-dl385pg8-08.lab.eng.pek2.redhat.com', RSA key 2048 bits, signed using RSA-SHA256, activated `2016-09-26 08:35:00 UTC', expires `2017-09-26 08:35:00 UTC', SHA-1 fingerprint `8fab8e2bb9e6f0f574f446f4b8703cf754afb513'
	Public Key ID:
		bb60a6027e76db2d1f5abb9efac529f8f64da123
	Public key's random art:
		+--[ RSA 2048]----+
		|                 |
		|                 |
		|                 |
		|                 |
		|        S    .   |
		|.       ... o .  |
		|..    +..E * .   |
		| ..o =.o=oB +    |
		|  o.o..+OXo. .   |
		+-----------------+

- Status: The certificate is trusted. 
- Successfully sent 1 certificate(s) to server.
- Description: (TLS1.2)-(ECDHE-RSA-SECP256R1)-(AES-128-GCM)
- Session ID: 0E:92:80:2D:13:36:3E:FC:66:5D:98:6C:17:AD:78:D0:A7:95:A6:9D:58:9E:5D:F2:80:1A:E4:AD:14:80:9E:A7
- Ephemeral EC Diffie-Hellman parameters
 - Using curve: SECP256R1
 - Curve size: 256 bits
- Version: TLS1.2
- Key Exchange: ECDHE-RSA
- Server Signature: RSA-SHA256
- Client Signature: RSA-SHA256
- Cipher: AES-128-GCM
- MAC: AEAD
- Compression: NULL
- Options: safe renegotiation,
- Handshake was completed

- Simple Client Mode:

                                                                               ^[[24;80R
Google, Inc.
Serial Graphics Adapter 12/29/13
SGABIOS $Id: sgabios.S 8 2010-04-22 00:03:40Z nlaredo $ (mockbuild@) Sun Dec 29 03:43:06 UTC 2013
4 0

 eaBIOS (version 1.9.1-4.el7)


iPXE (http://ipxe.org) 00:03.0 CA80 PCI2.10 PnP PMM+07F978B0+07EF78B0 CA80
                                                                               


Booting from Hard Disk...
Boot failed: could not read the boot disk

Booting from Floppy...
Boot failed: could not read the boot disk

Booting from DVD/CD...
Boot failed: Could not read from CDROM (code 0003)
Booting from ROM...
iPXE (PCI 00:03.0) starting execution...ok
iPXE initialising devices...ok



iPXE 1.0.0+ (6366fa7a) -- Open Source Network Boot Firmware -- http://ipxe.org
Features: DNS HTTP iSCSI TFTP AoE ELF MBOOT PXE bzImage Menu PXEXT
S
net0: 52:54:00:12:34:56 using 82540em on PCI00:03.0 (open)
  [Link:up, TX:0 TXE:0 RX:0 RXE:0]
Configuring (net0 52:54:00:12:34:56)............... ok
net0: 10.0.2.15/255.255.255.0 gw 10.0.2.2
Nothing to boot: No such file or directory (http://ipxe.org/2d03e13b)
No more network devices

No bootable device.  

Scenario 3,
Make sure the below keys are created under /home/Bug dir or your gnutls server dir
x509-client.pem -> client-cert.pem
x509-client-key.pem -> client-key.pem
1.boot up guest with cli
  /usr/libexec/qemu-kvm -m 8G -smp 16 -name vocado-vt-vm1 -vga std -device virtio-blk-pci,id=disk,drive=drive-disk0,bootindex=1 -drive file=RHEL-Server-7.3-64-virtio.raw,format=raw,if=none,id=drive-disk0,cache=none,werror=stop -vnc :1 -enable-kvm -monitor stdio -vnc :0 -object tls-creds-x509,id=tls0,endpoint=server,dir=/home/Bug -chardev socket,id=s0,host=hp-dl385pg8-08.lab.eng.pek2.redhat.com,port=12121,tls-creds=tls0,server,nowait -device isa-serial,chardev=s0 -device sga
2.login guest and execute "screen /dev/ttyS0"
3./usr/libexec/qemu-kvm -object tls-creds-x509,id=tls0,endpoint=client,dir=/home/Bug -chardev socket,id=s0,host=hp-dl385pg8-08.lab.eng.pek2.redhat.com,port=12121,tls-creds=tls0 -device isa-serial,chardev=s0 -display none -device sga

Expected results,the boot message pop up from terminal where you are executing screen /dev/ttyS0
Actual results,the boot message pop up from terminal where you are executing screen /dev/ttyS0


  After communicate with Daniel about results QE draw a conclusion,the bug has been fixed on build qemu-kvm-rhev-2.6.0-27.el7.x86_64.
  Thanks for Daniel's help !

Best regards,
Min

Comment 26 errata-xmlrpc 2016-11-07 20:56:30 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-2673.html