Bug 1300776

Summary: RFE: add support for native TLS encryption on chardev TCP transports
Product: Red Hat Enterprise Linux 7 Reporter: Daniel Berrangé <berrange>
Component: libvirtAssignee: John Ferlan <jferlan>
Status: CLOSED ERRATA QA Contact: yalzhang <yalzhang>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 7.3CC: berrange, dyuan, jdenemar, jferlan, mtessun, mzhan, rbalakri, virt-bugs, virt-maint, xuzhang, yafu, yalzhang, yanqzhan
Target Milestone: rcKeywords: FutureFeature
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: libvirt-2.5.0-1.el7 Doc Type: Enhancement
Doc Text:
Feature: Support TLS encryption on chardev TCP transport. Reason: Greater security configuration capabilities for the guest to communicate over TCP connections. Result: There are new configuration options in the /etc/libvirt/qemu.conf file 'chardev_tls_x509_cert_dir', 'chardev_tls_x509_verify', and 'chardev_tls_x509_secret_uuid' to allow for host level setup of the TLS environment to be used by any character device using a TCP backend. Additionally, a 'tls' property has been added to the TCP backend device definition. See http://libvirt.org/formatdomain.html#elementsCharTCP for some details about the tls property.
 Story Points: ---
Clone Of: 1300773 Environment:
Last Closed: 2017-08-01 17:09:12 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1300773    
Bug Blocks: 1301024    
Attachments:
Description Flags
The gnutls-serv output
none
vm log file for scenario 3 c#11 none

Description Daniel Berrangé 2016-01-21 17:00:15 UTC 
Clone of QEMU bug to track libvirt enablement tasks for native TLS encryption with TCP chardevs. 

+++ This bug was initially created as a clone of Bug #1300773 +++

Description of problem:
The character device TCP backend currently transfers all data in plain text, which offers no security protection, unless it is tunnelled over an external service like SSH. Such tunnelling is inefficient and adds management complexity. Thus it is desired to have native support for TLS encryption with the TCP backend for character devices.

Current patches for this are:

  https://lists.gnu.org/archive/html/qemu-devel/2016-01/msg03396.html
Comment 2 John Ferlan 2016-06-18 10:37:04 UTC 
Couple of recent series to go towards a solution:

TCP chardev related:

http://www.redhat.com/archives/libvir-list/2016-June/msg01094.html


Extraction and posting of patches related to bz 1301021 to add a new secret type (key) in order to handle the password for the TLS environment were posted here:

http://www.redhat.com/archives/libvir-list/2016-June/msg01051.html

More patches are ready to be posted in order to support the passwordid argument, but they require the key patches to be in first.
Comment 3 John Ferlan 2016-06-24 23:35:06 UTC 
Posted a v2 which includes everything that's necessary:

http://www.redhat.com/archives/libvir-list/2016-June/msg01709.html

First 6 patches from that series are in various states of being pushed as part of a different series (LUKS)
Comment 4 John Ferlan 2016-06-27 13:37:56 UTC 
Moving to 7.4
Comment 5 John Ferlan 2016-08-05 14:57:17 UTC 
Posted a v5 of the series:

http://www.redhat.com/archives/libvir-list/2016-August/msg00282.html
Comment 6 John Ferlan 2016-10-26 11:53:49 UTC 
After numerous more iterations and alterations to the code to adjust supporting algorithms:


v6: http://www.redhat.com/archives/libvir-list/2016-September/msg00302.html
v7: http://www.redhat.com/archives/libvir-list/2016-September/msg00729.html
v8: http://www.redhat.com/archives/libvir-list/2016-October/msg00306.html
v9: http://www.redhat.com/archives/libvir-list/2016-October/msg00726.html
v10: http://www.redhat.com/archives/libvir-list/2016-October/msg00866.html
v11: http://www.redhat.com/archives/libvir-list/2016-October/msg01069.html

Other "related" changes:
http://www.redhat.com/archives/libvir-list/2016-October/msg01023.html
http://www.redhat.com/archives/libvir-list/2016-October/msg00992.html
http://www.redhat.com/archives/libvir-list/2016-October/msg00976.html
http://www.redhat.com/archives/libvir-list/2016-October/msg00959.html
http://www.redhat.com/archives/libvir-list/2016-October/msg00958.html

The changes are now pushed upstream:


commit 8550e8585eef1ed7f5850a698d680e20b5cbcdff
Author: John Ferlan <jferlan>
Date:   Fri Jun 17 09:44:30 2016 -0400

    qemu: Add secret object hotplug for TCP chardev TLS
...

$ git describe 8550e8585eef1ed7f5850a698d680e20b5cbcdff
v2.3.0-206-g8550e85
$
Comment 8 yalzhang@redhat.com 2017-05-31 05:58:31 UTC 
Hi John,

Could you please help to confirm below questions:
1. set gnutls-serv as tls server, the guest console as client, the gnutls-serv showed the QEMU BIOS boot messages, but can not login from the console, if this is expected?

2. As the bug title said "native TLS encryption", it means the tls server and client should both on the same host, right?


Test on below packages
libvirt-3.2.0-6.el7.x86_64
qemu-kvm-rhev-2.9.0-7.el7.x86_64

1. ucomment below content in qemu.conf
chardev_tls = 1
chardev_tls_x509_cert_dir = "/etc/pki/libvirt-chardev"

2. prepare the certificates for server and client, and start the server.
# ll /etc/pki/libvirt-chardev/
total 16
-rw-r--r--. 1 root qemu 1086 May 31 13:30 ca-cert.pem
-r--r-----. 1 root qemu 1253 May 31 09:19 client-cert.pem
-r--r-----. 1 root qemu 5823 May 31 09:19 client-key.pem

# ll server* ca-cert.pem
-rw-r--r--. 1 root root 1086 May 31 13:22 ca-cert.pem
-rw-r--r--. 1 root root 1180 May 31 13:23 server-cert.pem
-r--------. 1 root root 5816 May 31 13:21 server-key.pem

# gnutls-serv    --echo --x509cafile ca-cert.pem /
                 --x509keyfile server-key.pem  /
                --x509certfile server-cert.pem 
Set static Diffie-Hellman parameters, consider --dhparams.
Processed 1 CA certificate(s).
Echo Server listening on IPv4 0.0.0.0 port 5556...done

3. The guest have os installed and "console=ttyS0,115200n" configured in the kernel command line. Start the guest with below serial device:

# virsh dumpxml rhel-new | grep /console -B9
    <serial type='tcp'>
      <source mode='connect' host='127.0.0.1' service='5556' tls='yes'/>
      <protocol type='raw'/>
      <target port='0'/>
    </serial>
    <console type='tcp'>
      <source mode='connect' host='127.0.0.1' service='5556' tls='yes'/>
      <protocol type='raw'/>
      <target type='serial' port='0'/>
    </console>

# virsh start rhel-new
Domain rhel-new started

4. check the gnutls-serv showed the QEMU BIOS boot messages as expected, but can not login from the console, refer to the attached messages, the output will end with below messages:

.......
*** Processing 63 bytes command: [  OK  ] Started dracut pre-pivot and cleanup hook.

received:          Starting Cleaning Up and Shutting Down Daemons...

*** Processing 61 bytes command:          Starting Cleaning Up and Shutting Down Daemons...

Scheduling inactive connection for close
Comment 9 yalzhang@redhat.com 2017-05-31 06:00:34 UTC 
Created attachment 1283607 [details]
The gnutls-serv output
Comment 10 yalzhang@redhat.com 2017-05-31 06:35:39 UTC 
Senario 2: 
Launch QEMU with a serial port as a TLS server. Run a generic gnutls-cli TLS client, it works as expected.

1. ucomment these lines in qemu.conf, and restart libvirtd service.
 chardev_tls = 1
 chardev_tls_x509_cert_dir = "/etc/pki/libvirt-chardev"
 chardev_tls_x509_verify = 1

2. start a vm with below content (the vm have os installed and console=ttyS0,115200 configured in kernel command line). The gnutls client will show bios boot messages as pretty format, and can login and execute commands.

# virsh dumpxml rhel-new | grep /console -B11
    <serial type='tcp'>
      <source mode='bind' host='server74' service='5556' tls='yes'/>
      <protocol type='raw'/>
      <target port='0'/>
      <alias name='serial0'/>
    </serial>
    <console type='tcp'>
      <source mode='bind' host='server74' service='5556' tls='yes'/>
      <protocol type='raw'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
# ps aux | grep qemu-kvm
...-object tls-creds-x509,id=objcharserial0_tls0,dir=/etc/pki/libvirt-chardev,endpoint=server,verify-peer=yes -chardev socket,id=charserial0,host=server74,port=5556,server,nowait,tls-creds=objcharserial0_tls0 -device isa-serial,chardev=charserial0,id=serial0 ...

# gnutls-cli --priority=NORMAL -p5556 --x509cafile=ca-cert.pem server74  --x509certfile=client-cert.pem --x509keyfile=client-key.pem
Processed 1 CA certificate(s).
Processed 1 client X.509 certificates...
Resolving 'server74'...
Connecting to '10.66.71.67:5556'...
- Certificate type: X.509
- Got a certificate list of 1 certificates.
- Certificate[0] info:
 - subject `CN=server74,O=libvirt.org', issuer `CN=libvirt.org', RSA key 2048 bits, signed using RSA-SHA256, activated `2017-05-31 00:33:30 UTC', expires `2018-05-31 00:33:30 UTC', SHA-1 fingerprint `d2002e2d40fe836aa1af642c9335fb6f18f543b1'
	Public Key ID:
		557b4983a9c15a1fb945991a8d068c60d148631c
	Public key's random art:
		+--[ RSA 2048]----+
		|     oE* +...B+o |
		|     oo.o =.Oo=o |
		|         o.=.*o  |
		|        ... +.   |
		|        S        |
		|                 |
		|                 |
		|                 |
		|                 |
		+-----------------+

- Status: The certificate is trusted. 
- Successfully sent 1 certificate(s) to server.
- Description: (TLS1.2)-(ECDHE-RSA-SECP256R1)-(AES-128-GCM)
- Session ID: 99:87:7A:CF:F8:2C:7A:B3:18:41:01:3B:AE:56:ED:A7:1F:1B:3B:53:93:8E:5D:97:02:DE:D4:CD:AE:D3:AE:3C
- Ephemeral EC Diffie-Hellman parameters
 - Using curve: SECP256R1
 - Curve size: 256 bits
- Version: TLS1.2
- Key Exchange: ECDHE-RSA
- Server Signature: RSA-SHA256
- Client Signature: RSA-SHA256
- Cipher: AES-128-GCM
- MAC: AEAD
- Compression: NULL
- Options: safe renegotiation,
- Handshake was completed

- Simple Client Mode:

[  OK  ] Started Show Plymouth Boot Screen.
[  OK  ] Reached target Paths.
[  OK  ] Reached target Basic System.
[  OK  ] Found device /dev/mapper/rhel-root.
         Starting File System Check on /dev/mapper/rhel-root...
[  OK  ] Started File System Check on /dev/mapper/rhel-root.
[  OK  ] Started dracut initqueue hook.
         Mounting /sysroot...
[  OK  ] Reached target Remote File Systems (Pre).
[  OK  ] Reached target Remote File Systems.
[  OK  ] Mounted /sysroot.
[  OK  ] Reached target Initrd Root File System.
         Starting Reload Configuration from the Real Root...
[  OK  ] Started Reload Configuration from the Real Root.
[  OK  ] Reached target Initrd File Systems.
[  OK  ] Reached target Initrd Default Target.
         Starting dracut pre-pivot and cleanup hook...
......

[  OK  ] Started Authorization Manager.
[  OK  ] Started Login Service.

Red Hat Enterprise Linux Server 7.4 Beta (Maipo)
Kernel 3.10.0-671.el7.x86_64 on an x86_64

localhost login: root
root
Password: redhat

Last login: Wed May 31 11:26:54 on tty1
[root@localhost ~]# ls
ls
anaconda-ks.cfg
[root@localhost ~]#
Comment 11 yalzhang@redhat.com 2017-05-31 08:00:09 UTC 
Scenario 3: 
Launch QEMU with a RHEL guest, and a serial port as a TLS server. Inside the guest OS run "tail -f /dev/ttyS0'. Then launch a second QEMU instance with a serial port as a TLS client, connecting to the first QEMU guest. The BIOS boot messages from the second QEMU should appear in the guest terminal where you're running 'tail -f /dev/ttyS0'.

1.
# ll /etc/pki/libvirt-chardev
total 28
-rw-r--r--. 1 root qemu 1086 May 31 13:30 ca-cert.pem
-r--r-----. 1 root qemu 1253 May 31 09:19 client-cert.pem
-r--r-----. 1 root qemu 5823 May 31 09:19 client-key.pem
-r--r-----. 1 root qemu 1180 May 31 14:06 server-cert.pem
-r--r-----. 1 root qemu 5826 May 31 14:06 server-key.pem

2. the 2 guests both have os installed and kernel command line configured with "console=ttyS0,115200"
# virsh dumpxml rhel7.4 | grep /console -B5
    </serial>
    <console type='tcp'>
      <source mode='bind' host='server74' service='5556' tls='yes'/>
      <protocol type='raw'/>
      <target type='serial' port='1'/>
    </console>

# virsh dumpxml rhel-new | grep /console -B5
    </serial>
    <console type='tcp'>
      <source mode='connect' host='server74' service='5556' tls='yes'/>
      <protocol type='raw'/>
      <target type='serial' port='0'/>
    </console>

3. start the server rhel7.4 and login the guest, then execute "tail -f /dev/ttyS0"

4. start the client rhel-new, the qemu command line looks like this:
qemu     19362 32.3  4.7 1954960 351096 ?      Sl   15:24   0:19 /usr/libexec/qemu-kvm -name guest=rhel7.4, ...... 
-object tls-creds-x509,id=objcharserial0_tls0,dir=/etc/pki/libvirt-chardev,endpoint=server,verify-peer=yes -chardev socket,id=charserial0,host=server74,port=5556,server,nowait,tls-creds=objcharserial0_tls0......

qemu     19460 44.7  3.0 1919036 223860 ?      Sl   15:25   0:07 /usr/libexec/qemu-kvm -name guest=rhel-new......
-object tls-creds-x509,id=objcharserial0_tls0,dir=/etc/pki/libvirt-chardev,endpoint=client,verify-peer=yes -chardev socket,id=charserial0,host=server74,port=5556,tls-creds=objcharserial0_tls0 ......

5. check the output in rhel7.4 
[root@localhost ~]# tail -f /dev/ttyS0


Welcome to Red Hat Enterprise Linux Server 7.4 Beta (Maipo)!
k on /dev/mapper/rhel-root.


<===The output is limited, 
or sometimes there is nothing output, but the log file shows "Handshake with parent is done" refer to the attached rhel-new.log file. If there is something need to be configured? what is the expected result? If it should provide login interface?
Comment 12 yalzhang@redhat.com 2017-05-31 08:13:19 UTC 
Created attachment 1283656 [details]
vm log file for scenario 3 c#11
Comment 13 yalzhang@redhat.com 2017-05-31 08:35:24 UTC 
(In reply to yalzhang from comment #11)
> Scenario 3: 
> Launch QEMU with a RHEL guest, and a serial port as a TLS server. Inside the
> guest OS run "tail -f /dev/ttyS0'. Then launch a second QEMU instance with a
> serial port as a TLS client, connecting to the first QEMU guest. The BIOS
> boot messages from the second QEMU should appear in the guest terminal where
> you're running 'tail -f /dev/ttyS0'.
......
> 5. check the output in rhel7.4 
> [root@localhost ~]# tail -f /dev/ttyS0
> 
> 
> Welcome to Red Hat Enterprise Linux Server 7.4 Beta (Maipo)!
> k on /dev/mapper/rhel-root.
> 
> 
> <===The output is limited, 
> or sometimes there is nothing output, but the log file shows "Handshake with
> parent is done" refer to the attached rhel-new.log file. If there is
> something need to be configured? what is the expected result? If it should
> provide login interface?

When I use "screen /dev/ttyS0", the bios boot messages can display on the rhel7.4 guest. And after the boot messages, there will be 
"[screen is terminating]" and the screen command return. So there is no login interface, right?
Comment 14 yalzhang@redhat.com 2017-06-02 03:53:10 UTC 
Hi Daniel, could you help to confirm the 2 questions in #c8 ? Thank you in advance~
Comment 15 yalzhang@redhat.com 2017-06-09 03:35:40 UTC 
1 more question:

1. Is the scenario 3 in c11 just for testing? as I found something strange, not sure if it is expected:

The boot message will show on the server guest when I run "screen /dev/ttyS0", this is as expected, but when I check on the client guest's screen, the boot message will show again and again, it can not boot into os. This is not related with the tls encryption.

one more scenario about migration:

 (1) migration succeed.
   <serial type="tcp">
    <source mode="connect" host="127.0.0.1" service="2445"/>
    <protocol type="telnet"/>
    <target port="1"/>
  </serial>

 (2) migration succeed.
   <serial type="tcp">
    <source mode="bind" host="127.0.0.1" service="2445"/>
    <protocol type="telnet"/>
    <target port="1"/>
  </serial>

 (3) migration succeed.
   <serial type="tcp">
    <source mode="connect" host="127.0.0.1" service="2445" tls='yes'/>
    <protocol type="telnet"/>
    <target port="1"/>
  </serial>

 (4) migration succeed.
  <serial type="tcp">
    <source mode="bind" host="127.0.0.1" service="2445" tls='yes'/>
    <protocol type="telnet"/>
    <target port="1"/>
  </serial>
Comment 16 yalzhang@redhat.com 2017-06-12 02:54:54 UTC 
one more scenario:
If no client-cert.pem for the vm's tls client endpoint, the boot messages will still shows on the guntls-serv. This is not expected, right?

1. on one terminal execute below command:
#  gnutls-serv    --echo --x509cafile ca-cert.pem  --x509keyfile server-key.pem  --x509certfile server-cert.pem 
Set static Diffie-Hellman parameters, consider --dhparams.
Processed 1 CA certificate(s).
Echo Server listening on IPv4 0.0.0.0 port 5556...done
Echo Server listening on IPv6 :: port 5556...done


2. rename ca-client to bak in /etc/pki/qemu:

# ll /etc/pki/qemu
total 24
-rw-r--r--. 1 root qemu 1253 Jun  9 11:26 bak
-rw-r--r--. 1 root qemu 1086 Jun  9 11:26 ca-cert.pem
-r--r--r--. 1 root qemu 5823 Jun  9 11:26 ca-key.pem
-r--r--r--. 1 root qemu 5823 Jun  9 11:26 client-key.pem

# ll /etc/pki/libvirt-chardev
ls: cannot access /etc/pki/libvirt-chardev: No such file or directory

3. prepare a guest with below content, and start

  <serial type='tcp'>
      <source mode='connect' host='127.0.0.1' service='5556' tls='yes'/>
      <protocol type='raw'/>
      <target port='0'/>
    </serial>
    <console type='tcp'>
      <source mode='connect' host='127.0.0.1' service='5556' tls='yes'/>
      <protocol type='raw'/>
      <target type='serial' port='0'/>
    </console>

# virsh start rhel7.4
Domain rhel7.4 started

4. check the output of gnutls-serv
# #  gnutls-serv    --echo --x509cafile ca-cert.pem  --x509keyfile server-key.pem  --x509certfile server-cert.pem 
Set static Diffie-Hellman parameters, consider --dhparams.
Processed 1 CA certificate(s).
Echo Server listening on IPv4 0.0.0.0 port 5556...done
Echo Server listening on IPv6 :: port 5556...done

* Accepted connection from IPv4 127.0.0.1 port 42228 on Mon Jun 12 10:52:02 2017
- Description: (TLS1.2)-(ECDHE-RSA-SECP256R1)-(AES-128-GCM)
- Session ID: 1A:58:48:67:D3:76:AA:02:AF:81:B4:B8:46:37:26:1F:99:05:34:51:75:1A:00:EE:92:CC:BA:36:8C:E7:C6:53
No certificates found!
- Ephemeral EC Diffie-Hellman parameters
 - Using curve: SECP256R1
 - Curve size: 256 bits
- Version: TLS1.2
- Key Exchange: ECDHE-RSA
- Server Signature: RSA-SHA256
- Cipher: AES-128-GCM
- MAC: AEAD
- Compression: NULL
- Options: safe renegotiation,
- Channel binding 'tls-unique': ddddd32e80ba4b8368ce3b54


-----boot messages ---------

received: [  OK  ] Stopped target Remote File Systems (Pre).

*** Processing 62 bytes command: [  OK  ] Stopped target Remote File Systems (Pre).

received: [  OK  ] Stopped dracut initqueue hook.

*** Processing 51 bytes command: [  OK  ] Stopped dracut initqueue hook.

received:          Stopping dracut initqueue hook...

*** Processing 45 bytes command:          Stopping dracut initqueue hook...

received: [  OK  ] Stopped target Initrd Default Target.

*** Processing 58 bytes command: [  OK  ] Stopped target Initrd Default Target.

Scheduling inactive connection for close
Comment 17 Daniel Berrangé 2017-06-15 10:05:13 UTC 
(In reply to yalzhang from comment #8)
> Hi John,
> 
> Could you please help to confirm below questions:
> 1. set gnutls-serv as tls server, the guest console as client, the
> gnutls-serv showed the QEMU BIOS boot messages, but can not login from the
> console, if this is expected?

The 'gnutls-serv' program is only able to display messages received from the client. It ignores any key presses in the gnutls-serv terminal and won't send them back to the client.  So you can't use gnutls-serv to login to the guest console - only view messages it sends. The fact that you can see the boot messages is sufficient to prove that TLS is working as expected.


> 2. As the bug title said "native TLS encryption", it means the tls server
> and client should both on the same host, right?

The client & server can be on separate hosts if desired - there's no restriction to only using "localhost"

> 4. check the gnutls-serv showed the QEMU BIOS boot messages as expected, but
> can not login from the console, refer to the attached messages, the output
> will end with below messages:
> 
> .......
> *** Processing 63 bytes command: [  OK  ] Started dracut pre-pivot and
> cleanup hook.
> 
> received:          Starting Cleaning Up and Shutting Down Daemons...
> 
> *** Processing 61 bytes command:          Starting Cleaning Up and Shutting
> Down Daemons...
> 
> Scheduling inactive connection for close

That is fine - it shows the boot messages which proves TLS is working.
Comment 18 Daniel Berrangé 2017-06-15 10:18:25 UTC 
(In reply to yalzhang from comment #16)
> one more scenario:
> If no client-cert.pem for the vm's tls client endpoint, the boot messages
> will still shows on the guntls-serv. This is not expected, right?

This depends on how you configure the server

> 1. on one terminal execute below command:
> #  gnutls-serv    --echo --x509cafile ca-cert.pem  --x509keyfile
> server-key.pem  --x509certfile server-cert.pem 

In this configuration, the server does *not* mandate any client cert. So it is expected that it'll still work with client-cert.pem doesn't exist.

If you give the arg --require-client-cert to gnutls-serv that'll mandate the client cert.

So that looks normal to me.
Comment 19 yalzhang@redhat.com 2017-06-15 11:20:14 UTC 
Thank you Daniel, set this bug to be verified.
Comment 20 errata-xmlrpc 2017-08-01 17:09:12 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1846
Comment 21 errata-xmlrpc 2017-08-01 23:51:16 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1846