Description of problem: I just copy&&paste Daniel's comment(https://bugzilla.redhat.com/show_bug.cgi?id=1724911#c15) here: "The default_tls_x509_verify setting does indeed appear to default to 0. This is likely inherited from when we generalized the previous vnc_tls_x509_verify option. This default was fine for VNC, as typically theres an authentication process applied on top of the TLS layer. For NBD/chardev/migration, this default is not a very good idea, as there's no extra auth layer. So this is a bug in libvirt defaults we should consider changing." Version-Release number of selected component (if applicable): libvirt-6.6.0-5.module+el8.3.0+8092+f9e72d7e.x86_64 How reproducible: 100% Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
Essentially with the default of 0, the client QEMU is validating that the server QEMU is authorized (by having its own server certificate). The server has no way to validate that the client is authorized, becuase it isn't providing a client certificate. This mutual authentication using certs is important when no other auth scheme exists for the protocol.
Fixed upstream: commit 6f30d7003a26cfd4e5fe9357abc9877cb5f393f3 Author: Peter Krempa <pkrempa> Date: Fri Nov 13 15:24:51 2020 +0100 NEWS: Mention change of default for TLS certificate verification Signed-off-by: Peter Krempa <pkrempa> Reviewed-by: Michal Privoznik <mprivozn> commit 0200fe42a09df06ad34432f603e03dfe7f345c41 Author: Peter Krempa <pkrempa> Date: Fri Nov 13 15:20:58 2020 +0100 qemu: conf: Enable 'backup_tls_x509_verify' by default The NBD server used to export pull-mode backups doesn't have any other form of client authentication on top of the TLS transport, so the only way to authenticate clients is to verify their certificate. Enable this option by defauilt when both 'backup_tls_x509_verify' and 'default_tls_x509_verify' were not configured. Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=1879477 Signed-off-by: Peter Krempa <pkrempa> Reviewed-by: Michal Privoznik <mprivozn> Reviewed-by: Eric Blake <eblake> commit 930583149c4701657a50686f200e475bd9e92a4a Author: Peter Krempa <pkrempa> Date: Fri Nov 13 15:20:58 2020 +0100 qemu: conf: Enable 'migrate_tls_x509_verify' by default The migration stream connection and also the NBD server for non-shared storage migration don't have any other form of client authentication on top of the TLS transport, so the only way to authenticate clients is to verify their certificate. Enable this option by defauilt when both 'migrate_tls_x509_verify' and 'default_tls_x509_verify' were not configured. Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=1879477 Signed-off-by: Peter Krempa <pkrempa> Reviewed-by: Michal Privoznik <mprivozn> Reviewed-by: Eric Blake <eblake> commit 019f962c86e32a9e3188d82013abd5d35cf8585c Author: Peter Krempa <pkrempa> Date: Fri Nov 13 15:20:58 2020 +0100 qemu: conf: Enable 'chardev_tls_x509_verify' by default Chardevs don't have any other form of client authentication on top of the TLS transport, so the only way to authenticate clients is to verify their certificate. Enable this option by defauilt when both 'chardev_tls_x509_verify' and 'default_tls_x509_verify' were not configured. Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=1879477 Signed-off-by: Peter Krempa <pkrempa> Reviewed-by: Michal Privoznik <mprivozn> commit 940ef34443946f2a92cc00195f4135aec822e348 Author: Peter Krempa <pkrempa> Date: Fri Nov 13 15:18:37 2020 +0100 qemu: conf: Clarify default of "vnc_tls_x509_verify" If both "vnc_tls_x509_verify" and "default_tls_x509_verify" are missing from the config file the client certificate validation is disabled. VNC provides a layer of authentication so client certificate validation is not strictly required. Signed-off-by: Peter Krempa <pkrempa> Reviewed-by: Michal Privoznik <mprivozn> commit 9ba2a06e47fbd09c6b2d44bd4ac6bf68781122ef Author: Peter Krempa <pkrempa> Date: Fri Nov 13 15:13:29 2020 +0100 qemu: conf: Allow individual control of default value for *_tls_x509_verify Store whether "default_tls_x509_verify" was provided and enhance the SET_TLS_VERIFY_DEFAULT macro so that indiviual users can provide their own default if "default_tls_x509_verify" config option was not provided. For now we keep setting it to 'false'. Signed-off-by: Peter Krempa <pkrempa> Reviewed-by: Michal Privoznik <mprivozn>
Verify with libvirt-6.10.0-1.module+el8.4.0+8898+a84e86e1.x86_64 ~~~~~~~~~~~~~~~~Test for migrate_tls_x509_verify: PASS~~~~~~~~~~~~~~~~~~~ Note: all the qemu.conf settings are on server host(migration target host) 1. Don't set default_tls_x509_verify or migrate_tls_x509_verify - PASS 1-1. don't create client cert, only create server cert 1) Migration with copy storage # virsh migrate rhel8-min qemu+ssh://fjin-8-4-av-rs/system --live --verbose --p2p --tls --copy-storage-all error: internal error: unable to execute QEMU command 'blockdev-add': Failed to read option reply: Cannot read from TLS channel: Software caused connection abort 2) Migration without copy storage # virsh migrate rhel8-min qemu+ssh://fjin-8-4-av/system --live --verbose --p2p --tls error: operation failed: migration out job: Cannot write to TLS channel: Input/output error 1-2. create both server cert and client cert 1) Migration with copy storage - succeed 2) Migration without copy storage - succeed 2. Don't set default_tls_x509_verify, set migrate_tls_x509_verify to 0 - PASS 2-1. don't create client cert, only create server cert 1) Migration with copy storage - succeed 2) Migration without copy storage - succeed 3. Set default_tls_x509_verify to 0, don't set migrate_tls_x509_verify - PASS 3-1. don't create client cert, only create server cert 1) Migration with copy storage - succeed 2) Migration without copy storage - succeed 4. Set default_tls_x509_verify to 0, set migrate_tls_x509_verify to 1 - PASS Same as scenario 1 5. Set default_tls_x509_verify to 1, set migrate_tls_x509_verify to 0 - PASS 5-1. don't create client cert, only create server cert 1) Migration with copy storage - succeed 2) Migration without copy storage - succeed
~~~~~~~~~~~~~~~~Test for chardev_tls_x509_verify: PASS~~~~~~~~~~~~~~~~~~~ 0. Prepare vm with tls chardev device: <serial type='tcp'> <source mode='bind' host='fjin-8-4-av' service='5556' tls='yes'/> <protocol type='raw'/> <target type='isa-serial' port='0'> <model name='isa-serial'/> </target> <alias name='serial0'/> </serial> 1. Don't set default_tls_x509_verify or chardev_tls_x509_verify - PASS 1-1. don't create client cert, only create server cert(connection fail) # gnutls-cli --priority=NORMAL -p5556 --x509cafile=ca-cert.pem fjin-8-4-av ... *** Fatal error: The TLS connection was non-properly terminated. *** Server has terminated the connection abnormally. 1-2. create both server cert and client cert(connection succeeds) # gnutls-cli --priority=NORMAL -p5556 --x509cafile=ca-cert.pem fjin-8-4-av --x509certfile=client-cert.pem --x509keyfile=client-key.pem 2. Don't set default_tls_x509_verify, set chardev_tls_x509_verify to 0 - PASS 2-1. don't create client cert, only create server cert(connection succeeds) # gnutls-cli --priority=NORMAL -p5556 --x509cafile=ca-cert.pem fjin-8-4-av 3. Set default_tls_x509_verify to 0, don't set chardev_tls_x509_verify - PASS same as 2-1 4. Set default_tls_x509_verify to 0, set chardev_tls_x509_verify to 1 - PASS Same as scenario 1 5. Set default_tls_x509_verify to 1, set chardev_tls_x509_verify to 0 - PASS same as 2-1
~~~~~~~~~~~~~~~~Test for backup_tls_x509_verify: PASS~~~~~~~~~~~~~~~~~~~ 0. Setup 0-1. Enable incremental backup(refer to: https://bugzilla.redhat.com/show_bug.cgi?id=1799015#c2) 0-2. Start a full backup: # cat backup.xml <domainbackup mode="pull"> <server name="fjin-8-4-av" port="10809" tls="yes"/> <disks> <disk backup="no" name="vda" /> <disk backup="yes" name="vdb" type="file"> <scratch file="/tmp/scratch_file_0" /> </disk> </disks> </domainbackup> # virsh backup-begin vm1 backup.xml Backup started 1. Don't set default_tls_x509_verify or backup_tls_x509_verify - PASS 1-1. don't create client cert, only create server cert(connection fail) # qemu-img convert -O qcow2 --object tls-creds-x509,id=tls0,endpoint=client,dir=/etc/pki/qemu/ 'json:{"file":{"driver":"nbd", "server":{"host":"fjin-8-4-av", "port":10809, "type":"inet"}, "export":"vdb", "tls-creds":"tls0"}}' test.qcow2 qemu-img: Could not open 'json:{"file":{"driver":"nbd", "server":{"host":"fjin-8-4-av", "port":10809, "type":"inet"}, "export":"vdb", "tls-creds":"tls0"}}': Failed to read option reply: Cannot read from TLS channel: Software caused connection abort 1-2. create both server cert and client cert(connection succeeds) # qemu-img convert -O qcow2 --object tls-creds-x509,id=tls0,endpoint=client,dir=/etc/pki/qemu/ 'json:{"file":{"driver":"nbd", "server":{"host":"fjin-8-4-av", "port":10809, "type":"inet"}, "export":"vdb", "tls-creds":"tls0"}}' test.qcow2 2. Don't set default_tls_x509_verify, set chardev_tls_x509_verify to 0 - PASS 2-1. don't create client cert, only create server cert(connection succeeds) # qemu-img convert -O qcow2 --object tls-creds-x509,id=tls0,endpoint=client,dir=/etc/pki/qemu/ 'json:{"file":{"driver":"nbd", "server":{"host":"fjin-8-4-av", "port":10809, "type":"inet"}, "export":"vdb", "tls-creds":"tls0"}}' test.qcow2 3. Set default_tls_x509_verify to 0, don't set chardev_tls_x509_verify - PASS same as 2-1 4. Set default_tls_x509_verify to 0, set chardev_tls_x509_verify to 1 - PASS Same as scenario 1 5. Set default_tls_x509_verify to 1, set chardev_tls_x509_verify to 0 - PASS same as 2-1
@yisun Please check comment6, I have done the test, please check whether the test cases need update, and comment in this BZ, then I can set qe_test_coverage field
@jinqi Please check comment5, I have done the test, please check whether the test cases need update, and comment in this BZ, then I can set qe_test_coverage field
(In reply to Fangge Jin from comment #7) > @yisun > Please check comment6, I have done the test, please check whether the test > cases need update, and comment in this BZ, then I can set qe_test_coverage > field Update related testcase, thx
(In reply to Fangge Jin from comment #8) > @jinqi > Please check comment5, I have done the test, please check whether the test > cases need update, and comment in this BZ, then I can set qe_test_coverage > field Test cases updated
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (virt:av bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2021:2098