Bug 1300772 - RFE: add support for native TLS encryption on NBD for disk migration
Summary: RFE: add support for native TLS encryption on NBD for disk migration
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: libvirt
Version: 7.3
Hardware: Unspecified
OS: Unspecified
Target Milestone: rc
: ---
Assignee: Peter Krempa
QA Contact: Han Han
Depends On: 1300770
Blocks: 1414999 1301025 ospd_block_based_live_migration 1544869 1625597 1664790 1665042
TreeView+ depends on / blocked
Reported: 2016-01-21 16:51 UTC by Daniel Berrangé
Modified: 2019-01-10 11:34 UTC (History)
14 users (show)

Fixed In Version: libvirt-4.4.0-1.el7
Doc Type: Enhancement
Doc Text:
Clone Of: 1300770
: 1544869 (view as bug list)
Last Closed: 2018-10-30 09:49:43 UTC
Target Upstream Version:

Attachments (Terms of Use)

Description Daniel Berrangé 2016-01-21 16:51:46 UTC
Clone of QEMU bug to track libvirt enablement tasks for native TLS encryption with NBD channel used during disk migration. 

+++ This bug was initially created as a clone of Bug #1300770 +++

Description of problem:
The NBD protocol currently runs in clear text, offering no security protection for the data transferred, unless it is tunnelled over some external transport like SSH. Such tunnelling is inefficient and inconvenient to manage, so there is a desire to add explicit support for TLS to the NBD clients & servers provided by QEMU.

A particular focus is on the need to have encryption of NBD channels used for disk copy during migration.

Latest patch series implementing TLS for NBD is


Comment 2 John Ferlan 2016-06-27 13:37:12 UTC
Moving to 7.4

Comment 3 Jaroslav Suchanek 2018-02-13 08:44:15 UTC
Reassigning to Peter as it relates to his blockdev-add work and has patches ready.

Comment 4 Peter Krempa 2018-05-23 11:30:41 UTC
The feature was added upstream with following commits (and a bunch of prequel refactors):

a8dc146a4d qemu: migration: Add support for transporting NBD over TLS
89ed82646c qemu: migration: Extract code responsible for calling drive-mirror
8bad8e7bf0 qemu: migration: Rename NBD migration functions
99223c8cca test: Add status XML test for NBD tls storage migration
93a3f9844f qemu: domain: Add private data for NBD migration storage source definition
35a8487ad9 qemu: domain: Extract parsing of NBD status XML
0d6ec712d3 qemu: domain: Extract NBD disk migration private data formatting
ceb151b24b qemu: block: Add helpers for hot-adding virStorageSource via blockdev
d30fd1fc54 qemu: monitor: Introduce support for blockdev-mirror
c7b66f2a59 qemu: monitor: Add implementation for blockdev-add and blockdev-del
a6178a6610 qemu: monitor: Factor out and document code to format QMP command
e969af4cec qemu: caps: Add capability for blockdev-add/blockdev-del
d4e57c6f04 qemu: migration: Don't access disk members without lock
082266bf27 conf: domain: Export virDomainStorageSourceParse
b8240fe704 qemu: block: Don't nest storage layer properties into format layer

Comment 6 Han Han 2018-07-06 09:33:13 UTC
Verified on libvirt-4.5.0-1.el7.x86_64 qemu-kvm-rhev-2.12.0-6.el7.x86_64:
I. Create tls cert on server (target host)
1. Set up a Certificate Authority (CA)
# certtool --generate-privkey > ca-key.pem

1.2 self-sign ca-key.pem by creating a file with the signature details called ca.info containing:
cn = $host1

# certtool --generate-self-signed --load-privkey ca-key.pem --template ca.info --outfile ca-cert.pem

2. Create server certificates
# certtool --generate-privkey > server-key.pem

2.2 sign that key with the CA's private key by first creating a template file called server.info:
organization = Red Hat
cn = $host1

# certtool --generate-certificate --load-privkey server-key.pem --load-ca-certificate ca-cert.pem \
--load-ca-privkey ca-key.pem --template server.info --outfile server-cert.pem

3. Copy CA key and server key to correct directory
3.1 # mkdir -p /etc/pki/qemu
3.2 # cp ca-key.pem ca-cert.pem /etc/pki/qemu
3.3 # cp server-key.pem /etc/pki/qemu
3.4 # cp server-cert.pem /etc/pki/qemu

II. Create client certs and keys
1. Copy CA cert to client(source host)

2.  Create client certificates

# certtool --generate-privkey > client-key.pem

Act as CA and sign the certificate.  Create client.info containing:

country = GB
state = London
locality = London
organization = Red Hat
cn = $host2

2.3 # certtool --generate-certificate  --load-privkey client-key.pem --load-ca-certificate /etc/pki/qemu/ca-cert.pem \
--load-ca-privkey /etc/pki/qemu/ca-key.pem --template client.info --outfile client-cert.pem

3. Copy client key to nbd tls directory

3.1 # cp client-key.pem /etc/pki/libvirt-nbd

3.2 # cp client-cert.pem /etc/pki/libvirt-nbd

Test nbd with tls:
1. Set qemu.conf:
nbd_tls = 1
nbd_tls_x509_cert_dir = "/etc/pki/libvirt-nbd"

2. Restart libvirtd

3. On server start qemu nbd:
# qemu-nbd --object tls-creds-x509,id=tls0,endpoint=server,dir=/root/server-keys --tls-creds=tls0 -p 10000 -t /tmp/c

4. On client host, prepare an VM with nbd disk:
# virsh dumpxml nbd-tls|grep disk -A 8
    <disk type='network' device='disk'>
      <driver name='qemu' type='raw'/>
      <source protocol='nbd'>
        <host name='lab.test.me' port='10000'/>
      <target dev='sda' bus='scsi'/>
      <alias name='ua-TLS'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>

Start VM, check qemu cmdline, there is tls object like following:
# ps aux|grep qemu|grep nbd
-object tls-creds-x509,id=objua-TLS_tls0,dir=/etc/pki/libvirt-nbd,endpoint=client,verify-peer=yes -drive file.driver=nbd,file.server.type=inet,file.server.host=lab.test.me,file.server.port=10000,file.tls-creds=objua-TLS_tls0,format=raw,if=none,id=drive-ua-TLS -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-ua-TLS,id=ua-TLS

Detech the nbd:
# virsh detach-disk nbd-tls sda
Disk detached successfully

Comment 7 Han Han 2018-07-09 05:47:17 UTC
Sorry,comment6 is for BZ1544869. Please ignore it.

Comment 8 Han Han 2018-08-21 06:25:11 UTC
Verified on libvirt-4.5.0-6.el7.x86_64 qemu-kvm-rhev-2.12.0-10.el7.x86_64:
1. Prepare two named hosts with libvirt and qemu-kvm-rhev installed
2. Open ports 49152-49215/tcp for the firewalls of hosts
3. Prepare a shareable storage, like glusterfs server
4. Prepare a VM.
5. Set tls env referring to https://bugzilla.redhat.com/show_bug.cgi?id=1300772#c6

Test steps
SC1: Tests migration cancel with --copy-storage-inc and --tls
1. Create backing file on src host and dst host:
# qemu-img create -f qcow2 -b 'json:{"file.driver":"gluster","file.volume":"gv0","file.path":"A.qcow2","file.server":[{"type":"inet","host":"XX.XX.XX.XX","port":"24007"}]}' /var/lib/libvirt/images/fuse.qcow2
Formatting '/var/lib/libvirt/images/fuse.qcow2', fmt=qcow2 size=10737418240 backing_file=json:{"file.driver":"gluster",,"file.volume":"gv0",,"file.path":"A.qcow2",,"file.server":[{"type":"
inet",,"host":"XX.XX.XX.XX",,"port":"24007"}]} cluster_size=65536 lazy_refcounts=off refcount_bits=16

2. Start VM with the backing file on disk and write some data in VM:
# virsh dumpxml fuse
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/fuse.qcow2'/>
      <backingStore type='network' index='1'>
        <format type='raw'/>
        <source protocol='gluster' name='gv0/A.qcow2'>
          <host name='XX.XX.XX.XX' port='24007'/>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0d' function='0x0'/>
(in VM) # dd if=/dev/urandom of=file bs=1G count=1
0+1 records in
0+1 records out
33554431 bytes (34 MB, 32 MiB) copied, 0.155817 s, 215 MB/s

3. Migrate with --tls --copy-storage-inc and cancel it before finish
# virsh migrate fuse qemu+ssh://root@test2.nbd/system  --tls --copy-storage-inc --verbose                                                                                
Migration: [ 32 %]^Cerror: operation aborted: migration out: canceled by client

SC2: Tests migration finish with --copy-storage-inc and --tls
1. Do as the steps in step1~2 in SC1
2. Migrate with --tls --copy-storage-inc
# virsh migrate fuse qemu+ssh://root@test2.nbd/system  --tls --copy-storage-inc --verbose
Migration: [100 %]

SC3: Tests migration cancel with --copy-storage-all and --tls
1. Prepare a local disk and an VM using it:
# qemu-img info /var/lib/libvirt/images/fuse.qcow2 -U                                    
image: /var/lib/libvirt/images/fuse.qcow2
file format: qcow2
virtual size: 10G (10737418240 bytes)
disk size: 1.4G
cluster_size: 65536
Format specific information:
    compat: 1.1
    lazy refcounts: false
    refcount bits: 16
    corrupt: false

# virsh dumpxml fuse
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/fuse.qcow2'/>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0d' function='0x0'/>

2. Migrate with --copy-storage-all --tls then cancel it:
# virsh migrate fuse qemu+ssh://root@test2.nbd/system  --tls --copy-storage-all --verbose                                                                                
Migration: [  9 %]^Cerror: operation aborted: migration out: canceled by client

SC4: Tests migration finish with --copy-storage-all and --tls
1. Do as step1 in SC3
2. Migrate with --copy-storage-all --tls:
# virsh migrate fuse qemu+ssh://root@test2.nbd/system  --tls --copy-storage-all --verbose
Migration: [100 %]

3.Do some write in VM
(In VM) # dd if=/dev/urandom of=file bs=100M count=2
dd: warning: partial read (33554431 bytes); suggest iflag=fullblock
0+2 records in
0+2 records out
67108862 bytes (67 MB) copied, 10.8148 s, 6.2 MB/s

All works well. Verified.

Comment 10 errata-xmlrpc 2018-10-30 09:49:43 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.