Hide Forgot
Clone of QEMU bug to track libvirt enablement tasks for native TLS encryption with NBD channel used during disk migration. +++ This bug was initially created as a clone of Bug #1300770 +++ Description of problem: The NBD protocol currently runs in clear text, offering no security protection for the data transferred, unless it is tunnelled over some external transport like SSH. Such tunnelling is inefficient and inconvenient to manage, so there is a desire to add explicit support for TLS to the NBD clients & servers provided by QEMU. A particular focus is on the need to have encryption of NBD channels used for disk copy during migration. Latest patch series implementing TLS for NBD is https://lists.gnu.org/archive/html/qemu-devel/2016-01/msg03440.html
Moving to 7.4
Reassigning to Peter as it relates to his blockdev-add work and has patches ready.
The feature was added upstream with following commits (and a bunch of prequel refactors): a8dc146a4d qemu: migration: Add support for transporting NBD over TLS 89ed82646c qemu: migration: Extract code responsible for calling drive-mirror 8bad8e7bf0 qemu: migration: Rename NBD migration functions 99223c8cca test: Add status XML test for NBD tls storage migration 93a3f9844f qemu: domain: Add private data for NBD migration storage source definition 35a8487ad9 qemu: domain: Extract parsing of NBD status XML 0d6ec712d3 qemu: domain: Extract NBD disk migration private data formatting ceb151b24b qemu: block: Add helpers for hot-adding virStorageSource via blockdev d30fd1fc54 qemu: monitor: Introduce support for blockdev-mirror c7b66f2a59 qemu: monitor: Add implementation for blockdev-add and blockdev-del a6178a6610 qemu: monitor: Factor out and document code to format QMP command e969af4cec qemu: caps: Add capability for blockdev-add/blockdev-del d4e57c6f04 qemu: migration: Don't access disk members without lock 082266bf27 conf: domain: Export virDomainStorageSourceParse b8240fe704 qemu: block: Don't nest storage layer properties into format layer
Verified on libvirt-4.5.0-1.el7.x86_64 qemu-kvm-rhev-2.12.0-6.el7.x86_64: Preparation: I. Create tls cert on server (target host) 1. Set up a Certificate Authority (CA) 1.1 # certtool --generate-privkey > ca-key.pem 1.2 self-sign ca-key.pem by creating a file with the signature details called ca.info containing: cn = $host1 ca cert_signing_key 1.3 # certtool --generate-self-signed --load-privkey ca-key.pem --template ca.info --outfile ca-cert.pem 2. Create server certificates 2.1 # certtool --generate-privkey > server-key.pem 2.2 sign that key with the CA's private key by first creating a template file called server.info: organization = Red Hat cn = $host1 tls_www_server encryption_key signing_key 2.3 # certtool --generate-certificate --load-privkey server-key.pem --load-ca-certificate ca-cert.pem \ --load-ca-privkey ca-key.pem --template server.info --outfile server-cert.pem 3. Copy CA key and server key to correct directory 3.1 # mkdir -p /etc/pki/qemu 3.2 # cp ca-key.pem ca-cert.pem /etc/pki/qemu 3.3 # cp server-key.pem /etc/pki/qemu 3.4 # cp server-cert.pem /etc/pki/qemu II. Create client certs and keys 1. Copy CA cert to client(source host) 2. Create client certificates 2.1 # certtool --generate-privkey > client-key.pem 2.2 Act as CA and sign the certificate. Create client.info containing: country = GB state = London locality = London organization = Red Hat cn = $host2 tls_www_client encryption_key signing_key 2.3 # certtool --generate-certificate --load-privkey client-key.pem --load-ca-certificate /etc/pki/qemu/ca-cert.pem \ --load-ca-privkey /etc/pki/qemu/ca-key.pem --template client.info --outfile client-cert.pem 3. Copy client key to nbd tls directory 3.1 # cp client-key.pem /etc/pki/libvirt-nbd 3.2 # cp client-cert.pem /etc/pki/libvirt-nbd Test nbd with tls: 1. Set qemu.conf: nbd_tls = 1 nbd_tls_x509_cert_dir = "/etc/pki/libvirt-nbd" 2. Restart libvirtd 3. On server start qemu nbd: # qemu-nbd --object tls-creds-x509,id=tls0,endpoint=server,dir=/root/server-keys --tls-creds=tls0 -p 10000 -t /tmp/c 4. On client host, prepare an VM with nbd disk: # virsh dumpxml nbd-tls|grep disk -A 8 <disk type='network' device='disk'> <driver name='qemu' type='raw'/> <source protocol='nbd'> <host name='lab.test.me' port='10000'/> </source> <target dev='sda' bus='scsi'/> <alias name='ua-TLS'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> Start VM, check qemu cmdline, there is tls object like following: # ps aux|grep qemu|grep nbd -object tls-creds-x509,id=objua-TLS_tls0,dir=/etc/pki/libvirt-nbd,endpoint=client,verify-peer=yes -drive file.driver=nbd,file.server.type=inet,file.server.host=lab.test.me,file.server.port=10000,file.tls-creds=objua-TLS_tls0,format=raw,if=none,id=drive-ua-TLS -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-ua-TLS,id=ua-TLS Detech the nbd: # virsh detach-disk nbd-tls sda Disk detached successfully
Sorry,comment6 is for BZ1544869. Please ignore it.
Verified on libvirt-4.5.0-6.el7.x86_64 qemu-kvm-rhev-2.12.0-10.el7.x86_64: Preparation: 1. Prepare two named hosts with libvirt and qemu-kvm-rhev installed 2. Open ports 49152-49215/tcp for the firewalls of hosts 3. Prepare a shareable storage, like glusterfs server 4. Prepare a VM. 5. Set tls env referring to https://bugzilla.redhat.com/show_bug.cgi?id=1300772#c6 Test steps SC1: Tests migration cancel with --copy-storage-inc and --tls 1. Create backing file on src host and dst host: # qemu-img create -f qcow2 -b 'json:{"file.driver":"gluster","file.volume":"gv0","file.path":"A.qcow2","file.server":[{"type":"inet","host":"XX.XX.XX.XX","port":"24007"}]}' /var/lib/libvirt/images/fuse.qcow2 Formatting '/var/lib/libvirt/images/fuse.qcow2', fmt=qcow2 size=10737418240 backing_file=json:{"file.driver":"gluster",,"file.volume":"gv0",,"file.path":"A.qcow2",,"file.server":[{"type":" inet",,"host":"XX.XX.XX.XX",,"port":"24007"}]} cluster_size=65536 lazy_refcounts=off refcount_bits=16 2. Start VM with the backing file on disk and write some data in VM: # virsh dumpxml fuse ... <disk type='file' device='disk'> <driver name='qemu' type='qcow2'/> <source file='/var/lib/libvirt/images/fuse.qcow2'/> <backingStore type='network' index='1'> <format type='raw'/> <source protocol='gluster' name='gv0/A.qcow2'> <host name='XX.XX.XX.XX' port='24007'/> </source> <backingStore/> </backingStore> <target dev='vda' bus='virtio'/> <alias name='virtio-disk0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x0d' function='0x0'/> </disk> ... (in VM) # dd if=/dev/urandom of=file bs=1G count=1 0+1 records in 0+1 records out 33554431 bytes (34 MB, 32 MiB) copied, 0.155817 s, 215 MB/s 3. Migrate with --tls --copy-storage-inc and cancel it before finish # virsh migrate fuse qemu+ssh://root/system --tls --copy-storage-inc --verbose Migration: [ 32 %]^Cerror: operation aborted: migration out: canceled by client SC2: Tests migration finish with --copy-storage-inc and --tls 1. Do as the steps in step1~2 in SC1 2. Migrate with --tls --copy-storage-inc # virsh migrate fuse qemu+ssh://root/system --tls --copy-storage-inc --verbose Migration: [100 %] SC3: Tests migration cancel with --copy-storage-all and --tls 1. Prepare a local disk and an VM using it: # qemu-img info /var/lib/libvirt/images/fuse.qcow2 -U image: /var/lib/libvirt/images/fuse.qcow2 file format: qcow2 virtual size: 10G (10737418240 bytes) disk size: 1.4G cluster_size: 65536 Format specific information: compat: 1.1 lazy refcounts: false refcount bits: 16 corrupt: false # virsh dumpxml fuse ... <disk type='file' device='disk'> <driver name='qemu' type='qcow2'/> <source file='/var/lib/libvirt/images/fuse.qcow2'/> <backingStore/> <target dev='vda' bus='virtio'/> <alias name='virtio-disk0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x0d' function='0x0'/> </disk> ... 2. Migrate with --copy-storage-all --tls then cancel it: # virsh migrate fuse qemu+ssh://root/system --tls --copy-storage-all --verbose Migration: [ 9 %]^Cerror: operation aborted: migration out: canceled by client SC4: Tests migration finish with --copy-storage-all and --tls 1. Do as step1 in SC3 2. Migrate with --copy-storage-all --tls: # virsh migrate fuse qemu+ssh://root/system --tls --copy-storage-all --verbose Migration: [100 %] 3.Do some write in VM (In VM) # dd if=/dev/urandom of=file bs=100M count=2 dd: warning: partial read (33554431 bytes); suggest iflag=fullblock 0+2 records in 0+2 records out 67108862 bytes (67 MB) copied, 10.8148 s, 6.2 MB/s All works well. Verified.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2018:3113