Description of problem: TLS can't be enalbed for the NBD server hosting the pull mode backup.
Implemented upstream: commit 423576679a5235ebb294bb9474f76abb7548e5e5 Author: Peter Krempa <pkrempa> Date: Wed Jul 1 12:38:42 2020 +0200 qemu: backup: Setup TLS environment for pull-mode backup jobs Use the configured TLS env to setup encryption of the TLS transport. https://bugzilla.redhat.com/show_bug.cgi?id=1822631 Signed-off-by: Peter Krempa <pkrempa> Reviewed-by: Eric Blake <eblake> commit fc6aaf6a194e6ee01cba3ee59e9fc3603189b77c Author: Peter Krempa <pkrempa> Date: Wed Jul 1 12:25:42 2020 +0200 conf: backup: Add 'tls' attribute for 'server' element Allow enabling TLS for the NBD server used to do pull-mode backups. Note that documentation already mentions 'tls', so this just implements the schema and XML bits. Signed-off-by: Peter Krempa <pkrempa> Reviewed-by: Eric Blake <eblake> commit d37a2cd5edcdc4aef4df09f5ca4f274a17072468 Author: Peter Krempa <pkrempa> Date: Fri Jun 26 16:37:16 2020 +0200 qemu: conf: Add configuration of TLS environment for NBD transport of pull-backups TLS is required to transport backed-up data securely when using pull-mode backups. Signed-off-by: Peter Krempa <pkrempa> Reviewed-by: Eric Blake <eblake> commit fffc147ba24aecd661fcb81725aa606386e88b88 Author: Peter Krempa <pkrempa> Date: Wed Jul 1 09:48:27 2020 +0200 conf: backup: Store 'tlsAlias' and 'tlsSecretAlias' as internals of a backup Add fields for storing the aliases necessary to clean up the TLS env for a backup job after it finishes. Signed-off-by: Peter Krempa <pkrempa> Reviewed-by: Eric Blake <eblake> commit bfd972167111c37f53bd8df1aa72474600dbed94 Author: Peter Krempa <pkrempa> Date: Wed Jul 1 09:56:46 2020 +0200 testCompareBackupXML: Add infrastructure for testing internal fields There are few internal fields of the backup XML. Propagate the 'internal' flag so that the test can verify the XML infrastructure. Signed-off-by: Peter Krempa <pkrempa> Reviewed-by: Eric Blake <eblake>
Test with libvirt-6.6.0-2.module+el8.3.0+7567+dc41c0a9.x86_64 Result is: PASS Test setup Following actions cwd is /etc/pki/libvirt-backup, which should be same as ‘backup_tls_x509_cert_dir’ in qemu.conf CA cert setup: 1. prepare a CA private key [root@dell-per730-67 libvirt-backup]# certtool --generate-privkey > ca-key.pem 2. prepare a CA info file [root@dell-per730-67 libvirt-backup]# cat ca.info cn = ca.redhat.com ca cert_signing_key 3. generate a CA cert signed by the private key [root@dell-per730-67 libvirt-backup]# certtool --generate-self-signed --load-privkey ca-key.pem --template ca.info --outfile ca-cert.pem Server side cert setup： 1. prepare a server private key [root@dell-per730-67 libvirt-backup]# certtool --generate-privkey > server-key.pem if we need to encrypt the private key, use following cmd: # certtool --password redhat --generate-privkey > server-key.pem 2. prepare a server info file [root@dell-per730-67 libvirt-backup]# cat server.info organization = Red Hat cn = dell-per730-67.lab.eng.pek2.redhat.com <=== server hostname tls_www_server encryption_key signing_key 3. generate a server cert If private key is encrypted, we’ll need to setup GUNTLS_PIN before generate certificate [root@dell-per730-67 libvirt-backup]# export GNUTLS_PIN=redhat [root@dell-per730-67 libvirt-backup]# certtool --generate-certificate --load-privkey server-key.pem --load-ca-certificate ca-cert.pem --load-ca-privkey ca-key.pem --template server.info --outfile server-cert.pem If private key is encrypted, we’ll need to setup GUNTLS_PIN before generate certificate [root@dell-per730-67 libvirt-backup]# export GNUTLS_PIN=redhat Client side cert setup： 1. prepare a client private key [root@dell-per730-67 libvirt-backup]# certtool --generate-privkey > client-key.pem Generating a 3072 bit RSA private key… 2. prepare a client info file [root@dell-per730-67 libvirt-backup]# cat client.info country = GB state = London locality = London organization = Red Hat cn = dell-per730-67.lab.eng.pek2.redhat.com <=== client hostname tls_www_client encryption_key signing_key 3. generate a client cert [root@dell-per730-67 libvirt-backup]# certtool --generate-certificate --load-privkey client-key.pem --load-ca-certificate ca-cert.pem --load-ca-privkey ca-key.pem --template client.info --outfile client-cert.pem Generating a signed certificate… Test tools Script to dump data for incremental backup - ‘get_inc_backup.sh’ This script is to get the data from the nbd export, and dump it to a local image. [root@dell-per730-67 ~]# cat get_inc_backup.sh if test $# -lt 6 || test $# -gt 7; then echo 'usage: copyif nbd_server nbd_export nbd_port target_img bitmap tls_dir' return 1 fi NBD_SERVER=$1 NBD_EXPORT=$2 NBD_PORT=$3 TARGET_IMG=$4 BITMAP=$5 TLS_DIR=$6 map_from="--image-opts driver=nbd,export=vdb,server.type=inet" map_from+=",server.host=$NBD_SERVER,server.port=10809,tls-creds=tls0" map_from+=",x-dirty-bitmap=qemu:dirty-bitmap:$BITMAP" state=false echo $map_from qemu-img rebase -u -f qcow2 -F raw -b 'json:{"file":{"driver":"nbd", "server":{"host":"'$NBD_SERVER'", "port":'$NBD_PORT', "type":"inet"}, "export":"'$NBD_EXPORT'", "tls-creds":"tls0"}}' $TARGET_IMG while read line; do [[ $line =~ .*start.:.([0-9]*).*length.:.([0-9]*).*data.:.$state.* ]] || continue start=${BASH_REMATCH[1]} len=${BASH_REMATCH[2]} echo echo " $start $len:" qemu-io -C -c "r $start $len" -f qcow2 $TARGET_IMG --object tls-creds-x509,id=tls0,endpoint=client,dir=$TLS_DIR done < <(qemu-img map --object tls-creds-x509,id=tls0,endpoint=client,dir=$TLS_DIR --output=json $map_from) qemu-img rebase -u -f qcow2 -b '' $TARGET_IMG Usage: Vm has a existing checkpoint [root@dell-per730-67 ~]# virsh checkpoint-list vm1 Name Creation Time ------------------------------------------- checkpoint_1 2020-08-17 04:16:20 -0400 Prepare a backup xml [root@dell-per730-67 ~]# cat bkup_inc1.xml <domainbackup mode="pull"> <incremental>checkpoint_1</incremental> <server name="dell-per730-67.lab.eng.pek2.redhat.com" port="10809" tls="yes"/> <disks> <disk backup="no" name="vda" /> <disk backup="yes" name="vdb" type="file"> <scratch file="/tmp/scratch_file_0" /> </disk> </disks> </domainbackup> Start the backup [root@dell-per730-67 ~]# virsh backup-begin vm1 bkup_inc1.xml Backup started Prepare the target image [root@dell-per730-67 ~]# qemu-img create -f qcow2 vdb.inc1.backup 1G Formatting 'vdb.inc1.backup', fmt=qcow2 cluster_size=65536 compression_type=zlib size=1073741824 lazy_refcounts=off refcount_bits=16 Run the script [root@dell-per730-67 ~]# ./get_inc_backup.sh dell-per730-67.lab.eng.pek2.redhat.com vdb 10809 vdb.inc1.backup backup-vdb /etc/pki/libvirt-backup/ --image-opts driver=nbd,export=vdb,server.type=inet,server.host=dell-per730-67.lab.eng.pek2.redhat.com,server.port=10809,tls-creds=tls0,x-dirty-bitmap=qemu:dirty-bitmap:backup-vdb 20971520 128974848: read 128974848/128974848 bytes at offset 20971520 123 MiB, 1 ops; 00.53 sec (229.993 MiB/sec and 1.8699 ops/sec) Test scenarios backup with default backup_tls_x509_cert_dir Positive test - Full backup with default backup_tls_x509_cert_dir Prepare a vdb for vm [root@dell-per730-67 ~]# qemu-img create -f qcow2 /var/lib/libvirt/images/vdb.qcow2 1G Formatting '/var/lib/libvirt/images/vdb.qcow2', fmt=qcow2 cluster_size=65536 compression_type=zlib size=1073741824 lazy_refcounts=off refcount_bits=16 [root@dell-per730-67 ~]# virsh domblklist vm1 Target Source -------------------------------------------------------- ... vdb /var/lib/libvirt/images/vdb.qcow2 Make sure comment out following params in qemu.conf: # backup_tls_x509_cert_dir = "/etc/pki/libvirt-backup" # backup_tls_x509_verify = 1 # backup_tls_x509_secret_uuid = "00000000-0000-0000-0000-000000000000" Restart libvirtd daemon [root@dell-per730-67 ~]# systemctl restart libvirtd Generate 123MB data in vm’s vdb [root@dell-per730-67 ~]# virsh start vm1 --console [root@localhost ~]# dd if=/dev/urandom of=/dev/vdb bs=1M seek=100 count=123; sync Start a full backup [root@dell-per730-67 ~]# cat backup_pull_0.xml <domainbackup mode="pull"><server name="dell-per730-67.lab.eng.pek2.redhat.com" port="10809" tls="yes"/><disks><disk backup="no" name="vda" /><disk backup="yes" name="vdb" type="file"><scratch file="/tmp/scratch_file_0" /></disk></disks></domainbackup> [root@dell-per730-67 ~]# virsh backup-begin vm1 backup_pull_0.xml Backup started Dump the backup data to a local image [root@dell-per730-67 ~]# qemu-img convert -O qcow2 --object tls-creds-x509,id=tls0,endpoint=client,dir=/etc/pki/libvirt-backup/ 'json:{"file":{"driver":"nbd", "server":{"host":"dell-per730-67.lab.eng.pek2.redhat.com", "port":10809, "type":"inet"}, "export":"vdb", "tls-creds":"tls0"}}' test.qcow2 Check the new image has 123MB data [root@dell-per730-67 ~]# qemu-img info test.qcow2 image: test.qcow2 file format: qcow2 virtual size: 1 GiB (1073741824 bytes) disk size: 123 MiB cluster_size: 65536 Format specific information: compat: 1.1 compression type: zlib lazy refcounts: false refcount bits: 16 corrupt: false backup with custom backup_tls_x509_cert_dir Positive test - Incremental backup with custom backup_tls_x509_cert_dir Mkdir /etc/pki/test and set backup tls dir to it [root@dell-per730-67 pki]# mkdir /etc/pki/test [root@dell-per730-67 pki]# cat /etc/libvirt/qemu.conf | grep backup_tls_x509_cert_dir backup_tls_x509_cert_dir = "/etc/pki/test" [root@dell-per730-67 ~]# systemctl restart libvirtd Prepare full backup&checkpoint xml [root@dell-per730-67 ~]# cat backup_full.xml <domainbackup mode="pull"> <server name="dell-per730-67.lab.eng.pek2.redhat.com" port="10809" tls="yes"/> <disks> <disk backup="no" name="vda" /> <disk backup="yes" name="vdb" type="file"> <scratch file="/tmp/scratch_file_0" /> </disk> </disks> </domainbackup> [root@dell-per730-67 ~]# cat check_0.xml <domaincheckpoint> <disks> <disk checkpoint="no" name="vda" /> <disk checkpoint="bitmap" name="vdb" /> </disks> <name>checkpoint_0</name> <description>desc of cp_0</description> </domaincheckpoint> [root@dell-per730-67 ~]# cat backup_inc.xml <domainbackup mode="pull"> <incremental>checkpoint_0</incremental> <server name="dell-per730-67.lab.eng.pek2.redhat.com" port="10809" tls="yes"/> <disks> <disk backup="no" name="vda" /> <disk backup="yes" name="vdb" type="file"> <scratch file="/tmp/scratch_file_0" /> </disk> </disks> </domainbackup> [root@dell-per730-67 ~]# cat check_1.xml <domaincheckpoint> <disks> <disk checkpoint="no" name="vda" /> <disk checkpoint="bitmap" name="vdb" /> </disks> <name>checkpoint_1</name> <description>desc of cp_1</description> </domaincheckpoint> Generate 123MB data in vm’s vdb [root@dell-per730-67 ~]# virsh console vm1 [root@localhost ~]# dd if=/dev/urandom of=/dev/vdb bs=1M seek=20 count=123; sync Start full backup without preparing any certs [root@dell-per730-67 ~]# virsh backup-begin vm1 backup_full.xml check_0.xml error: internal error: unable to execute QEMU command 'object-add': Unable to access credentials /etc/pki/test/ca-cert.pem: No such file or directory ⇐ failed as expected [root@dell-per730-67 ~]# virsh checkpoint-list vm1 Name Creation Time ----------------------- ⇐ no checkpoint created Prepare ca+server key/certs in /etc/pki/test/ [root@dell-per730-67 pki]# ll /etc/pki/test total 131056 -rw-r--r--. 1 root root 1448 Aug 18 01:45 ca-cert.pem -rw-r--r--. 1 root root 8177 Aug 18 01:45 ca-key.pem -rw-r--r--. 1 root root 39 Aug 18 01:45 ca.info -rw-r--r--. 1 root root 1639 Aug 18 01:45 client-cert.pem -rw-r--r--. 1 root root 8177 Aug 18 01:45 client-key.pem -rw-r--r--. 1 root root 155 Aug 18 01:45 client.info -rw-r--r--. 1 root root 1578 Aug 18 01:45 server-cert.pem -rw-r--r--. 1 root root 8180 Aug 18 01:45 server-key.pem -rw-r--r--. 1 root root 109 Aug 18 01:45 server.info Start the full backup [root@dell-per730-67 ~]# virsh backup-begin vm1 backup_full.xml check_0.xml Backup started [root@dell-per730-67 ~]# virsh checkpoint-list vm1 setlocale: No such file or directory Name Creation Time ------------------------------------------- checkpoint_0 2020-08-18 01:46:26 -0400 ⇐ checkpoint_0 created [root@dell-per730-67 ~]# virsh domjobabort vm1 Generate 321MB data in vm1’s vdb [root@localhost ~]# dd if=/dev/urandom of=/dev/vdb bs=1M seek=500 count=321; sync Start the incremental backup (from checkpoint_0) [root@dell-per730-67 ~]# virsh backup-begin vm1 backup_inc.xml check_1.xml Backup started Generate incremental backup data to a local image [root@dell-per730-67 ~]# qemu-img create -f qcow2 vdb.inc.backup 1G Formatting 'vdb.inc.backup', fmt=qcow2 cluster_size=65536 compression_type=zlib size=1073741824 lazy_refcounts=off refcount_bits=16 [root@dell-per730-67 ~]# ./get_inc_backup.sh dell-per730-67.lab.eng.pek2.redhat.com vdb 10809 vdb.inc.backup backup-vdb /etc/pki/test/ --image-opts driver=nbd,export=vdb,server.type=inet,server.host=dell-per730-67.lab.eng.pek2.redhat.com,server.port=10809,tls-creds=tls0,x-dirty-bitmap=qemu:dirty-bitmap:backup-vdb 524288000 336592896: read 336592896/336592896 bytes at offset 524288000 321 MiB, 1 ops; 0:00:01.35 (238.349 MiB/sec and 0.7425 ops/sec) [root@dell-per730-67 ~]# qmeu-img info vdb.inc.backup -bash: qmeu-img: command not found [root@dell-per730-67 ~]# qemu-img info vdb.inc.backup image: vdb.inc.backup file format: qcow2 virtual size: 1 GiB (1073741824 bytes) disk size: 321 MiB ⇐ the disk size is 321MB which is same as the incremental data we generated after full backup cluster_size: 65536 Format specific information: compat: 1.1 compression type: zlib lazy refcounts: false refcount bits: 16 corrupt: false Full backup with backup_tls_x509_secret_uuid Env preparation Prepare the server cert: Prepare CA key+cert as before Prepare server private key: [root@dell-per730-67 libvirt-backup]# certtool --password redhat --generate-privkey > server-key.pem Assuming PKCS #8 format... Generating a 3072 bit RSA private key... Set the local variable GNUTLS_PIN [root@dell-per730-67 libvirt-backup]# export GNUTLS_PIN=redhat Generate the server’s signed cert [root@dell-per730-67 libvirt-backup]# certtool --generate-certificate --load-privkey server-key.pem --load-ca-certificate ca-cert.pem --load-ca-privkey ca-key.pem --template server.info --outfile server-cert.pem Prepare libvirt secret: Prepare secret xml [root@dell-per730-67 libvirt-backup]# cat secret.xml <secret ephemeral='no' private='no'> <uuid>e3ba1f0c-01dc-4edd-97e5-1f5edb93b50d</uuid> <description>TLS passphrase</description> <usage type='tls'> <name>backup_tls_secret</name> </usage> </secret> Define the secret [root@dell-per730-67 ~]# virsh secret-define secret.xml Positive test - use a libvirt secret with correct passphrase to do the backup Set backup_tls_x509_secret_uuid to the libvirt secret uuid we just prepared [root@dell-per730-67 ~]# cat /etc/libvirt/qemu.conf | grep 'backup_tls_x509_secret_uuid =' backup_tls_x509_secret_uuid = "e3ba1f0c-01dc-4edd-97e5-1f5edb93b50d" Set password=redhat to the secret [root@dell-per730-67 ~]# MYSECRET=`printf %s "redhat" | base64` [root@dell-per730-67 ~]# virsh secret-set-value e3ba1f0c-01dc-4edd-97e5-1f5edb93b50d $MYSECRET error: Passing secret value as command-line argument is insecure! Secret value set Generate 111MB data in vm’s vdb [root@dell-per730-67 ~]# virsh start vm1 --console ... [root@localhost ~]# dd if=/dev/urandom of=/dev/vdb bs=1M seek=20 count=111; sync Start the backup [root@dell-per730-67 ~]# cat backup_full.xml <domainbackup mode="pull"> <server name="dell-per730-67.lab.eng.pek2.redhat.com" port="10809" tls="yes"/> <disks> <disk backup="no" name="vda" /> <disk backup="yes" name="vdb" type="file"> <scratch file="/tmp/scratch_file_0" /> </disk> </disks> </domainbackup> [root@dell-per730-67 ~]# virsh backup-begin vm1 backup_full.xml Backup started Dump the backup data to a local image [root@dell-per730-67 ~]# qemu-img convert -O qcow2 --object tls-creds-x509,id=tls0,endpoint=client,dir=/etc/pki/libvirt-backup 'json:{"file":{"driver":"nbd", "server":{"host":"dell-per730-67.lab.eng.pek2.redhat.com", "port":10809, "type":"inet"}, "export":"vdb", "tls-creds":"tls0"}}' test.qcow2 [root@dell-per730-67 ~]# qemu-img info test.qcow2 image: test.qcow2 file format: qcow2 virtual size: 1 GiB (1073741824 bytes) disk size: 111 MiB ⇐ the disk size is 111MB, same as the data we generated to vm’s vdb cluster_size: 65536 Format specific information: compat: 1.1 compression type: zlib lazy refcounts: false refcount bits: 16 corrupt: false Abort the backup job [root@dell-per730-67 ~]# virsh domjobabort vm1 Negative test - use libvirt secret with incorrect passphrase to do the backup Set the libvirt secret with a wrong password [root@dell-per730-67 ~]# MYSECRET=`printf %s "greenhat" | base64` [root@dell-per730-67 ~]# virsh secret-set-value e3ba1f0c-01dc-4edd-97e5-1f5edb93b50d $MYSECRET error: Passing secret value as command-line argument is insecure! Secret value set Try to start the backup job [root@dell-per730-67 ~]# virsh backup-begin vm1 backup_full.xml error: internal error: unable to execute QEMU command 'object-add': Cannot load certificate '/etc/pki/libvirt-backup/server-cert.pem' & key '/etc/pki/libvirt-backup/server-key.pem': Decryption has failed. ⇐ failed as expected Negative test - disable backup_tls_x509_secret_uuid and do the backup Not provide backup_tls_x509_secret_uuid [root@dell-per730-67 ~]# cat /etc/libvirt/qemu.conf | grep 'backup_tls_x509_secret_uuid =' # backup_tls_x509_secret_uuid = "e3ba1f0c-01dc-4edd-97e5-1f5edb93b50d" ⇐ it’s commented out Restart libvirtd daemon [root@dell-per730-67 ~]# systemctl restart libvirtd [root@dell-per730-67 ~]# virsh backup-begin vm1 backup_full.xml error: internal error: unable to execute QEMU command 'object-add': Cannot load certificate '/etc/pki/libvirt-backup/server-cert.pem' & key '/etc/pki/libvirt-backup/server-key.pem': Decryption has failed. ⇐ failed as expected Test backup_tls_x509_verify param in qemu.conf Negative test - set ‘backup_tls_x509_verify = 1’ and not providing the ca-cert.pem Prepare qemu.conf as follow and restart libvirtd [root@dell-per730-67 pki]# cat /etc/libvirt/qemu.conf | grep "backup_tls_x509_verify =" backup_tls_x509_verify = 1 [root@dell-per730-67 pki]# systemctl restart libvirtd Move the ca-cert.pem to tmp dir [root@dell-per730-67 pki]# mv /etc/pki/libvirt-backup/ca-cert.pem /tmp/ Start the backup, error happens as expected [root@dell-per730-67 ~]# virsh backup-begin vm1 backup_full.xml error: internal error: unable to execute QEMU command 'object-add': Unable to access credentials /etc/pki/libvirt-backup/ca-cert.pem: No such file or directory Positive test - set ‘backup_tls_x509_verify = 0’ and not providing the ca-cert.pem Prepare qemu.conf as follow and restart libvirtd [root@dell-per730-67 pki]# cat /etc/libvirt/qemu.conf | grep "backup_tls_x509_verify =" backup_tls_x509_verify = 0 [root@dell-per730-67 pki]# systemctl restart libvirtd Move the ca-cert.pem to tmp dir [root@dell-per730-67 pki]# mv /etc/pki/libvirt-backup/ca-cert.pem /tmp/ [root@dell-per730-67 ~]# virsh backup-begin vm1 backup_full.xml error: internal error: unable to execute QEMU command 'object-add': Unable to access credentials /etc/pki/libvirt-backup/ca-cert.pem: No such file or directory ⇐ not as expected ? Other corner cases Set tls=’yes’ for unix socket pull mode backup Prepare a backup xml as follow: [root@dell-per730-67 ~]# cat backup_unix.xml <domainbackup mode="pull"> <server socket="/tmp/pull_backup.socket" transport="unix" tls="yes"/> <disks> <disk backup="no" name="vda" /> <disk backup="yes" name="vdb" type="file"> <scratch file="/tmp/scratch_file_1" /> </disk> </disks> </domainbackup> Start the backup [root@dell-per730-67 ~]# virsh backup-begin vm1 backup_unix.xml error: internal error: unable to execute QEMU command 'nbd-server-start': TLS is only supported with IPv4/IPv6 ⇐ error msg is clear
well, the format of above comment is a little ugly... Same test info can be found at: https://docs.google.com/document/d/1DaFtumX6M9XOl-n1Y3aiICbjbT49QqCAQDKcD2id1F4
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (virt:8.3 bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:5137