Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1602328

Summary: [RFE] Add support for encrypted TLS client keys for disks
Product: Red Hat Enterprise Linux Advanced Virtualization Reporter: Peter Krempa <pkrempa>
Component: libvirtAssignee: Peter Krempa <pkrempa>
Status: CLOSED ERRATA QA Contact: Han Han <hhan>
Severity: medium Docs Contact:
Priority: medium    
Version: 8.0CC: dyuan, jdenemar, jsuchane, lmen, mtessun, xuzhang, yalzhang
Target Milestone: rcKeywords: FutureFeature, Triaged
Target Release: 8.1Flags: knoel: mirror+
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: libvirt-6.6.0-1.el8 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-11-17 17:44:45 UTC Type: Feature Request
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Peter Krempa 2018-07-18 08:09:09 UTC 
Description of problem:
Libvirt currently does not support providing of the secret for decrypting the TLS client key for disks.
Comment 2 Jaroslav Suchanek 2019-04-24 12:40:37 UTC 
This bug is going to be addressed in the next major release.
Comment 3 Peter Krempa 2020-07-07 11:20:56 UTC 
Fixed upstream:

commit ab7b3167b383b2af9bace432c17b5301ce4b8bc2
Author: Peter Krempa <pkrempa>
Date:   Tue Jun 30 17:08:44 2020 +0200

    tests: qemuxml2argv: Test encrypted TLS key for nbd/vxhs disks
    
    Add a dummy secret so that we see what command line is generated.
    
    Signed-off-by: Peter Krempa <pkrempa>
    Reviewed-by: Eric Blake <eblake>

commit 423711aef266377f1bfc5557ad8a364b0b03a4da
Author: Peter Krempa <pkrempa>
Date:   Tue Jun 30 15:39:41 2020 +0200

    qemu: domain: Setup secret for TLS key for nbd/vxhs disks
    
    Setup the TLS secret when preparing a virStorageSource for use.
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1602328
    
    Signed-off-by: Peter Krempa <pkrempa>
    Reviewed-by: Eric Blake <eblake>

commit 470d322e0cc3d45eccf54c6e4fa13bf1bd40428f
Author: Peter Krempa <pkrempa>
Date:   Mon Jun 29 17:12:03 2020 +0200

    qemu: conf: Add configuration of TLS key encryption for 'vxhs' and 'nbd' disks
    
    Until now libvirt didn't allow using encrypted TLS key for disk clients.
    
    Add fields for configuring the secret and propagate defaults.
    
    Signed-off-by: Peter Krempa <pkrempa>
    Reviewed-by: Eric Blake <eblake>

commit 7fa772bfd719ca8cf07ec5482266572fd21d6f5d
Author: Peter Krempa <pkrempa>
Date:   Mon Jun 29 15:11:00 2020 +0200

    qemu block: Add internals for handling 'secret' corresponding to TLS key
    
    Add infrastructure for hot- and cold-plug of the secret object holding
    decryption key for the TLS key.
    
    Signed-off-by: Peter Krempa <pkrempa>
    Reviewed-by: Eric Blake <eblake>

commit a3cbbc5afb4a9ad447c42776063351274eb2b7a6
Author: Peter Krempa <pkrempa>
Date:   Mon Jun 29 15:10:42 2020 +0200

    qemu: domain: Add infrastructure passing in TLS key's decryption key via 'secret'
    
    Store the required data in the private data of a storage source and
    ensure that the 'alias' of the secret is formatted in the status XML.
    
    Signed-off-by: Peter Krempa <pkrempa>
    Reviewed-by: Eric Blake <eblake>

commit cd0dc1c19feab350a235185cdf38d21d6c155216
Author: Peter Krempa <pkrempa>
Date:   Tue Jun 30 17:56:08 2020 +0200

    virQEMUDriverConfigLoadSpecificTLSEntry: Split up fetching of server-only config options
    
    The '*_tls_x509_verify' options are relevant only when we are going to
    expose a server socket as client sockets always enable verification.
    
    Split up the macro to separate the common bits from the server bits so
    that when we'll later extend support of 'nbd' and 'vxhs' disks which are
    client only we can reuse the existing macros.
    
    Signed-off-by: Peter Krempa <pkrempa>
    Reviewed-by: Eric Blake <eblake>

commit 94b5e9ebf62be0292cda9b3c31f2be1e385316cb
Author: Peter Krempa <pkrempa>
Date:   Tue Jun 30 17:51:50 2020 +0200

    virQEMUDriverConfigLoadSpecificTLSEntry: Move fetching of 'chardev_tls' above macro
    
    Move the extraction of the config value so that it makes more sense
    after upcoming refactors.
    
    Signed-off-by: Peter Krempa <pkrempa>
    Reviewed-by: Eric Blake <eblake>

commit 473b97abac5f4cc898f397be12a8fbcc0302a36a
Author: Peter Krempa <pkrempa>
Date:   Mon Jun 29 17:05:35 2020 +0200

    qemu: conf: Move 'nbd' and 'vxhs' tls config variables together with rest of tls setup
    
    Signed-off-by: Peter Krempa <pkrempa>
    Reviewed-by: Eric Blake <eblake>

commit 72fdba7ffd82e7f4c44ea239d001c51a54fb6285
Author: Peter Krempa <pkrempa>
Date:   Mon Jun 29 12:05:03 2020 +0200

    qemu.conf: Remove misleading mention of 'migrate_tls'
    
    There's no such parameter. Reword the sentence to account for enabling
    TLS-encrypted migration using API flags.
    
    Signed-off-by: Peter Krempa <pkrempa>
    Reviewed-by: Eric Blake <eblake>

commit 7bfb85cf1d138c4cee92a776bcda267468304a43
Author: Peter Krempa <pkrempa>
Date:   Mon Jun 29 16:00:03 2020 +0200

    qemuDomainDiskHasEncryptionSecret: unexport
    
    Signed-off-by: Peter Krempa <pkrempa>
    Reviewed-by: Eric Blake <eblake>

commit 47be72571915e3b920dce316a8375987c8c44223
Author: Peter Krempa <pkrempa>
Date:   Fri May 22 12:29:10 2020 +0200

    qemu: domain: Introduce helper for always fetching virStorageSource private data
    
    Add a helper which will always return the storage source private data
    even if it was not allocated before.
    
    Signed-off-by: Peter Krempa <pkrempa>
    Reviewed-by: Eric Blake <eblake>
Comment 4 Han Han 2020-07-21 08:00:11 UTC 
Test on v6.5.0-234-gc4cd0ef917 qemu-kvm-core-5.0.0-2.module+el8.3.0+7379+0505d6ca.x86_64 but FAILED:
Libvirt compiling options:
../configure --without-libssh --build=x86_64-redhat-linux-gnu --host=x86_64-redhat-linux-gnu --program-prefix= --disable-dependency-tracking --prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/etc --datadir=/usr/share --includedir=/usr/include --libdir=/usr/lib64 --libexecdir=/usr/libexec --localstatedir=/var --sharedstatedir=/var/lib --mandir=/usr/share/man --infodir=/usr/share/info --with-qemu --without-openvz --without-lxc --without-vbox --without-libxl --with-sasl --with-polkit --with-libvirtd --without-phyp --with-esx --without-hyperv --without-vmware --without-xenapi --without-vz --without-bhyve --without-interface --with-network --with-storage-fs --with-storage-lvm --with-storage-iscsi --with-storage-iscsi-direct --with-storage-scsi --with-storage-disk --with-storage-mpath --without-storage-sheepdog --with-storage-gluster --without-storage-zfs --without-storage-vstorage --with-numactl --with-numad --with-capng --without-fuse --without-netcf --with-selinux --without-apparmor --without-hal --with-udev --with-yajl --without-sanlock --with-libpcap --with-macvtap --with-audit --with-dtrace --with-driver-modules --with-firewalld --with-firewalld-zone --without-wireshark-dissector --without-pm-utils --with-nss-plugin '--with-packager=Unknown, 2019-08-19-12:13:01, lab.rhel8.me' --with-packager-version=1.el8 --with-qemu-user=qemu --with-qemu-group=qemu --with-tls-priority=@LIBVIRT,SYSTEM --enable-werror --enable-expensive-tests --with-init-script=systemd --without-login-shell


Setup:
1. Generate keys and certs of CA, server, client. For client keys, encrypted with passphrase "redhat"
2. Copy all keys and certs to /etc/pki/libvirt-nbd/
3. Prepare a image /tmp/test. Prepare the nbd tls server:
# qemu-nbd --object tls-creds-x509,id=tls0,endpoint=server,dir=/etc/pki/libvirt-nbd  /tmp/test -x new -t -v


Steps:
1. Prepare /etc/libvirt/qemu.conf and /etc/libvirt/libvirtd.conf :
qemu.conf:
nbd_tls = 1
nbd_tls_x509_cert_dir = "/etc/pki/libvirt-nbd"
max_core = "unlimited"
nbd_tls_x509_secret_uuid = "567e84ae-e792-4a74-b22d-f01445281862"

libvirtd.conf:
log_level = 3
log_filters = "1:qemu 4:object 4:json"
log_outputs = "1:file:/tmp/libvirtd.log"

2. Start libvirtd and virtlogd 
# LD_LIBRARY_PATH="./src/.libs" ./src/.libs/libvirtd
# LD_LIBRARY_PATH="./src/.libs" ./src/.libs/virtlogd

3. Define a libvirt tls secret with uuid 567e84ae-e792-4a74-b22d-f01445281862
Set the password to "redhat":
# virsh secret-set-value 567e84ae-e792-4a74-b22d-f01445281862 `printf redhat|base64`

4. Start a vm with nbd tls disk
Disk xml:
    <disk type='network' device='disk'>
      <driver name='qemu' type='raw'/>
      <source protocol='nbd' name='new' tls='yes'>
        <host name='XXXX' port='10809'/>
      </source>
      <target dev='sdb' bus='scsi'/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>

# virsh start fedora31                                                                                                                                                                                                 
error: Failed to start domain fedora31
error: internal error: process exited while connecting to monitor: 2020-07-21T07:57:33.126804Z qemu-kvm: -blockdev {"driver":"nbd","server":{"type":"inet","host":"XXXX","port":"10809"},"export":"new","tls-creds":"objlibvirt-1-storage_tls0","node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"}: Denied by server for option 5 (starttls)
server reported: TLS not configured

However the qemu-img info with passphrase object works:
# qemu-img info --object secret,id=sec,data=redhat,format=raw --object tls-creds-x509,id=tls0,endpoint=client,dir=/etc/pki/libvirt-nbd/,verify-peer=yes,passwordid=sec nbd://XXXX/new
image: nbd://XXXX:10809/new
file format: raw
virtual size: 100 MiB (104857600 bytes)
disk size: unavailable


The qemu log:
-chardev spicevmc,id=charredir1,name=usbredir \
-device usb-redir,chardev=charredir1,id=redir1,bus=usb.0,port=3 \
-device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x8 \
-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
-msg timestamp=on
2020-07-21 07:54:21.749+0000: Domain id=2 is tainted: host-cpu
char device redirected to /dev/pts/23 (label charserial0)
2020-07-21T07:54:21.916845Z qemu-kvm: -blockdev {"driver":"nbd","server":{"type":"inet","host":"XXXX","port":"10809"},"export":"new","tls-creds":"objlibvirt-1-storage_tls0","node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"}: Denied by server for option 5 (starttls)
server reported: TLS not configured
2020-07-21 07:54:21.969+0000: shutting down, reason=failed
2020-07-21 07:57:32.953+0000: starting up libvirt version: 6.6.0, package: 1.el8 (Unknown, 2019-08-19-12:13:01, lab.rhel8.me), qemu version: 5.0.50qemu-kvm-5.0.0-2.module+el8.3.0+7379+0505d6ca, kernel: 4.18.0-221.el8.x86_64, hostname: XXXX
LC_ALL=C \
LD_LIBRARY_PATH=./src/.libs \
PATH=/root/.gem/ruby/2.4.0/bin:/root/.cargo/bin:/root/.local/bin:/root/.gem/ruby/2.4.0/bin:/root/.cargo/bin:/root/.local/bin:/usr/share/Modules/bin:/usr/lib64/ccache:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin:/root/bin \
HOME=/var/lib/libvirt/qemu/domain-3-fedora31 \
USER=root \
LOGNAME=root \
XDG_DATA_HOME=/var/lib/libvirt/qemu/domain-3-fedora31/.local/share \
XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain-3-fedora31/.cache \
XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain-3-fedora31/.config \
QEMU_AUDIO_DRV=spice \
/usr/libexec/qemu-kvm \
-name guest=fedora31,debug-threads=on \
-S \
-object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-3-fedora31/master-key.aes \
-machine pc-i440fx-rhel7.6.0,accel=kvm,usb=off,vmport=off,dump-guest-core=off \
-cpu host,migratable=on \
-m 1024 \
-overcommit mem-lock=off \
-smp 1,sockets=1,cores=1,threads=1 \
-uuid 9e49557d-7326-4cfc-8d37-6207bb17e7f5 \
-no-user-config \
-nodefaults \
-chardev socket,id=charmonitor,fd=29,server,nowait \
-mon chardev=charmonitor,id=monitor,mode=control \
-rtc base=utc,driftfix=slew \
-global kvm-pit.lost_tick_policy=delay \
-no-hpet \
-no-shutdown \
-global PIIX4_PM.disable_s3=1 \
-global PIIX4_PM.disable_s4=1 \
-boot strict=on \
-device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x5.0x7 \
-device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x5 \
-device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x5.0x1 \
-device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x5.0x2 \
-device virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x6 \
-device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x7 \
-blockdev '{"driver":"file","filename":"/var/lib/libvirt/images/disk.qcow2","node-name":"libvirt-2-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-2-format","read-only":false,"driver":"qcow2","file":"libvirt-2-storage","backing":null}' \
-device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,device_id=drive-scsi0-0-0-0,drive=libvirt-2-format,id=scsi0-0-0-0,bootindex=1 \
-object secret,id=objlibvirt-1-storage_tls0-secret0,data=Et94qHmL96TI59qIrqOsDw==,keyid=masterKey0,iv=/a/LFpyIaJudfSbiEUdq5w==,format=base64 \
-object tls-creds-x509,id=objlibvirt-1-storage_tls0,dir=/etc/pki/libvirt-nbd,endpoint=client,verify-peer=yes,passwordid=objlibvirt-1-storage_tls0-secret0 \
-blockdev '{"driver":"nbd","server":{"type":"inet","host":"XXXX","port":"10809"},"export":"new","tls-creds":"objlibvirt-1-storage_tls0","node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-1-format","read-only":false,"driver":"raw","file":"libvirt-1-storage"}' \
-device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=1,device_id=drive-scsi0-0-0-1,drive=libvirt-1-format,id=scsi0-0-0-1 \
-netdev tap,fd=34,id=hostnet0 \
-device e1000,netdev=hostnet0,id=net0,mac=52:54:00:b0:15:11,bus=pci.0,addr=0x3 \
-chardev pty,id=charserial0 \
-device isa-serial,chardev=charserial0,id=serial0 \
-chardev spicevmc,id=charchannel0,name=vdagent \
-device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0 \
-device usb-tablet,id=input0,bus=usb.0,port=1 \
-spice port=0,disable-ticketing,image-compression=off,seamless-migration=on \
-device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vram64_size_mb=0,vgamem_mb=16,max_outputs=1,bus=pci.0,addr=0x2 \
-device intel-hda,id=sound0,bus=pci.0,addr=0x4 \
-device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 \
-chardev spicevmc,id=charredir0,name=usbredir \
-device usb-redir,chardev=charredir0,id=redir0,bus=usb.0,port=2 \
-chardev spicevmc,id=charredir1,name=usbredir \
-device usb-redir,chardev=charredir1,id=redir1,bus=usb.0,port=3 \
-device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x8 \
-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
-msg timestamp=on
2020-07-21 07:57:32.953+0000: Domain id=3 is tainted: host-cpu
char device redirected to /dev/pts/24 (label charserial0)
2020-07-21T07:57:33.126804Z qemu-kvm: -blockdev {"driver":"nbd","server":{"type":"inet","host":"XXXX","port":"10809"},"export":"new","tls-creds":"objlibvirt-1-storage_tls0","node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"}: Denied by server for option 5 (starttls)
server reported: TLS not configured
2020-07-21 07:57:33.176+0000: shutting down, reason=failed


Libvirtd log:
2020-07-21 07:54:21.994+0000: 1600129: debug : qemuDomainObjEndAsyncJob:1168 : Stopping async job: start (vm=0x7fcbd00deae0 name=fedora31)
2020-07-21 07:54:21.994+0000: 1617872: debug : qemuProcessHandleMonitorEOF:295 : Received EOF on 0x7fcbd00deae0 'fedora31'
2020-07-21 07:54:21.994+0000: 1617872: debug : qemuMonitorDispose:225 : mon=0x7fcc180430b0
2020-07-21 07:54:21.995+0000: 1600323: debug : qemuProcessEventHandler:4898 : vm=0x7fcbd00deae0, event=7
2020-07-21 07:54:21.995+0000: 1600323: debug : qemuProcessKill:7227 : vm=0x7fcbd00deae0 name=fedora31 pid=-1 flags=0x1
2020-07-21 07:54:21.995+0000: 1600323: debug : qemuProcessKill:7231 : VM 'fedora31' not active
2020-07-21 07:54:21.995+0000: 1600323: debug : qemuDomainObjBeginJobInternal:830 : Starting job: job=destroy agentJob=none asyncJob=none (vm=0x7fcbd00deae0 name=fedora31, current job=none agentJob=none async=none)
2020-07-21 07:54:21.995+0000: 1600323: debug : qemuDomainObjBeginJobInternal:877 : Started job: destroy (async=none vm=0x7fcbd00deae0 name=fedora31)
2020-07-21 07:54:21.995+0000: 1600323: debug : processMonitorEOFEvent:4790 : Domain 0x7fcbd00deae0 'fedora31' is not active, ignoring EOF
2020-07-21 07:54:21.995+0000: 1600323: debug : qemuDomainObjEndJob:1130 : Stopping job: destroy (async=none vm=0x7fcbd00deae0 name=fedora31)
2020-07-21 07:57:32.871+0000: 1600130: debug : qemuDomainObjBeginJobInternal:830 : Starting job: job=none agentJob=none asyncJob=start (vm=0x7fcbd00deae0 name=fedora31, current job=none agentJob=none async=none)
2020-07-21 07:57:32.871+0000: 1600130: debug : qemuDomainObjBeginJobInternal:885 : Started async job: start (vm=0x7fcbd00deae0 name=fedora31)
2020-07-21 07:57:32.871+0000: 1600130: debug : qemuProcessStart:7086 : conn=0x7fcc1c005aa0 driver=0x7fcbd003d710 vm=0x7fcbd00deae0 name=fedora31 id=-1 asyncJob=start migrateFrom=<null> migrateFd=-1 migratePath=<null> snapshot=(nil) vmop=0 flags=0x1
2020-07-21 07:57:32.871+0000: 1600130: debug : qemuProcessInit:5517 : vm=0x7fcbd00deae0 name=fedora31 id=-1 migration=0
2020-07-21 07:57:32.871+0000: 1600130: debug : qemuProcessInit:5519 : Beginning VM startup process
2020-07-21 07:57:32.871+0000: 1600130: debug : qemuProcessInit:5537 : Determining emulator version
2020-07-21 07:57:32.883+0000: 1600130: debug : virQEMUCapsCacheLookup:5644 : Returning caps 0x7fcbd0026250 for /usr/libexec/qemu-kvm
2020-07-21 07:57:32.883+0000: 1600130: debug : qemuProcessStartValidate:5326 : Checking for KVM availability
2020-07-21 07:57:32.883+0000: 1600130: debug : qemuProcessStartValidate:5337 : Checking domain and device security labels
2020-07-21 07:57:32.883+0000: 1600130: debug : virQEMUCapsCacheLookup:5644 : Returning caps 0x7fcbd0026250 for /usr/libexec/qemu-kvm
2020-07-21 07:57:32.883+0000: 1600130: debug : virQEMUCapsCacheLookup:5644 : Returning caps 0x7fcbd0026250 for /usr/libexec/qemu-kvm
2020-07-21 07:57:32.884+0000: 1600130: debug : virQEMUCapsCacheLookup:5644 : Returning caps 0x7fcbd0026250 for /usr/libexec/qemu-kvm
2020-07-21 07:57:32.884+0000: 1600130: debug : virQEMUCapsCacheLookup:5644 : Returning caps 0x7fcbd0026250 for /usr/libexec/qemu-kvm
2020-07-21 07:57:32.884+0000: 1600130: debug : virQEMUCapsCacheLookup:5644 : Returning caps 0x7fcbd0026250 for /usr/libexec/qemu-kvm
2020-07-21 07:57:32.884+0000: 1600130: debug : virQEMUCapsCacheLookup:5644 : Returning caps 0x7fcbd0026250 for /usr/libexec/qemu-kvm
2020-07-21 07:57:32.884+0000: 1600130: debug : virQEMUCapsCacheLookup:5644 : Returning caps 0x7fcbd0026250 for /usr/libexec/qemu-kvm
2020-07-21 07:57:32.884+0000: 1600130: debug : virQEMUCapsCacheLookup:5644 : Returning caps 0x7fcbd0026250 for /usr/libexec/qemu-kvm
2020-07-21 07:57:32.884+0000: 1600130: debug : virQEMUCapsCacheLookup:5644 : Returning caps 0x7fcbd0026250 for /usr/libexec/qemu-kvm
2020-07-21 07:57:32.884+0000: 1600130: debug : virQEMUCapsCacheLookup:5644 : Returning caps 0x7fcbd0026250 for /usr/libexec/qemu-kvm
2020-07-21 07:57:32.884+0000: 1600130: debug : virQEMUCapsCacheLookup:5644 : Returning caps 0x7fcbd0026250 for /usr/libexec/qemu-kvm
2020-07-21 07:57:32.885+0000: 1600130: debug : virQEMUCapsCacheLookup:5644 : Returning caps 0x7fcbd0026250 for /usr/libexec/qemu-kvm
2020-07-21 07:57:32.885+0000: 1600130: debug : virQEMUCapsCacheLookup:5644 : Returning caps 0x7fcbd0026250 for /usr/libexec/qemu-kvm
2020-07-21 07:57:32.885+0000: 1600130: debug : virQEMUCapsCacheLookup:5644 : Returning caps 0x7fcbd0026250 for /usr/libexec/qemu-kvm
2020-07-21 07:57:32.885+0000: 1600130: debug : virQEMUCapsCacheLookup:5644 : Returning caps 0x7fcbd0026250 for /usr/libexec/qemu-kvm
2020-07-21 07:57:32.885+0000: 1600130: debug : virQEMUCapsCacheLookup:5644 : Returning caps 0x7fcbd0026250 for /usr/libexec/qemu-kvm
2020-07-21 07:57:32.885+0000: 1600130: debug : virQEMUCapsCacheLookup:5644 : Returning caps 0x7fcbd0026250 for /usr/libexec/qemu-kvm
2020-07-21 07:57:32.885+0000: 1600130: debug : virQEMUCapsCacheLookup:5644 : Returning caps 0x7fcbd0026250 for /usr/libexec/qemu-kvm
2020-07-21 07:57:32.885+0000: 1600130: debug : virQEMUCapsCacheLookup:5644 : Returning caps 0x7fcbd0026250 for /usr/libexec/qemu-kvm
2020-07-21 07:57:32.885+0000: 1600130: debug : virQEMUCapsCacheLookup:5644 : Returning caps 0x7fcbd0026250 for /usr/libexec/qemu-kvm
2020-07-21 07:57:32.886+0000: 1600130: debug : virQEMUCapsCacheLookup:5644 : Returning caps 0x7fcbd0026250 for /usr/libexec/qemu-kvm
2020-07-21 07:57:32.886+0000: 1600130: debug : virQEMUCapsCacheLookup:5644 : Returning caps 0x7fcbd0026250 for /usr/libexec/qemu-kvm
2020-07-21 07:57:32.886+0000: 1600130: debug : virQEMUCapsCacheLookup:5644 : Returning caps 0x7fcbd0026250 for /usr/libexec/qemu-kvm
2020-07-21 07:57:32.886+0000: 1600130: debug : qemuProcessStartValidate:5392 : Checking for any possible (non-fatal) issues
2020-07-21 07:57:32.886+0000: 1600130: debug : qemuProcessInit:5551 : Setting current domain def as transient
2020-07-21 07:57:32.887+0000: 1600130: debug : qemuDomainAssignVirtioSerialAddresses:136 : Finished reserving existing ports
2020-07-21 07:57:32.887+0000: 1600130: debug : qemuDomainUSBAddressAddHubs:3034 : Found 3 USB devices and 6 provided USB ports; adding 0 hubs
2020-07-21 07:57:32.887+0000: 1600130: debug : qemuDomainAssignUSBAddresses:3173 : Existing USB addresses have been reserved
2020-07-21 07:57:32.887+0000: 1600130: debug : qemuDomainAssignUSBAddresses:3181 : Finished assigning USB ports
2020-07-21 07:57:32.887+0000: 1600130: debug : qemuProcessPrepareDomain:6203 : Generating domain security label (if required)
2020-07-21 07:57:32.888+0000: 1600130: debug : qemuProcessPrepareDomain:6233 : Assigning domain PCI addresses
2020-07-21 07:57:32.888+0000: 1600130: debug : qemuDomainAssignVirtioSerialAddresses:136 : Finished reserving existing ports
2020-07-21 07:57:32.888+0000: 1600130: debug : qemuDomainUSBAddressAddHubs:3034 : Found 3 USB devices and 6 provided USB ports; adding 0 hubs
2020-07-21 07:57:32.888+0000: 1600130: debug : qemuDomainAssignUSBAddresses:3173 : Existing USB addresses have been reserved
2020-07-21 07:57:32.888+0000: 1600130: debug : qemuDomainAssignUSBAddresses:3181 : Finished assigning USB ports
2020-07-21 07:57:32.888+0000: 1600130: debug : qemuProcessPrepareDomain:6242 : Setting graphics devices
2020-07-21 07:57:32.888+0000: 1600130: debug : qemuProcessPrepareDomain:6246 : Create domain masterKey
2020-07-21 07:57:32.888+0000: 1600130: debug : qemuProcessPrepareDomain:6250 : Setting up storage
2020-07-21 07:57:32.889+0000: 1600130: debug : qemuProcessPrepareDomain:6254 : Prepare chardev source backends for TLS
2020-07-21 07:57:32.889+0000: 1600130: debug : qemuProcessPrepareDomain:6257 : Prepare device secrets
2020-07-21 07:57:32.889+0000: 1600130: debug : qemuProcessPrepareDomain:6261 : Prepare bios/uefi paths
2020-07-21 07:57:32.889+0000: 1600130: debug : qemuProcessPrepareDomain:6267 : Preparing external devices
2020-07-21 07:57:32.889+0000: 1600130: debug : qemuProcessPrepareDomain:6280 : Preparing monitor state
2020-07-21 07:57:32.889+0000: 1600130: debug : qemuProcessPrepareDomain:6289 : Updating guest CPU definition
2020-07-21 07:57:32.889+0000: 1600130: debug : qemuProcessPrepareHost:6466 : Preparing network devices
2020-07-21 07:57:32.890+0000: 1600130: debug : qemuProcessPrepareHost:6471 : Preparing host devices
2020-07-21 07:57:32.890+0000: 1600130: debug : qemuProcessPrepareHost:6480 : Preparing chr devices
2020-07-21 07:57:32.890+0000: 1600130: debug : qemuProcessPrepareHost:6492 : Ensuring no historical cgroup is lying around
2020-07-21 07:57:32.905+0000: 1600130: debug : qemuProcessPrepareHost:6525 : Write domain masterKey
2020-07-21 07:57:32.913+0000: 1600130: debug : qemuProcessPrepareHost:6529 : Preparing disks (host)
2020-07-21 07:57:32.919+0000: 1600130: debug : qemuProcessPrepareHost:6533 : Preparing external devices
2020-07-21 07:57:32.919+0000: 1600130: debug : qemuProcessLaunch:6672 : conn=0x7fcc1c005aa0 driver=0x7fcbd003d710 vm=0x7fcbd00deae0 name=fedora31 if=3 asyncJob=6 incoming.launchURI=<null> incoming.deferredURI=<null> incoming.fd=-1 incoming.path=<null> snapshot=(nil) vmop=0 flags=0x11
2020-07-21 07:57:32.919+0000: 1600130: debug : qemuProcessLaunch:6702 : Creating domain log file
2020-07-21 07:57:32.919+0000: 1600130: debug : qemuDomainLogContextNew:6048 : Context new 0x7fcc20047000 stdioLogD=1
2020-07-21 07:57:32.920+0000: 1600130: debug : qemuProcessLaunch:6718 : Building emulator command line
2020-07-21 07:57:32.920+0000: 1600130: debug : qemuBuildCommandLine:9889 : driver=0x7fcbd003d710 def=0x7fcc28042a80 mon=0x7fcc20046060 qemuCaps=0x7fcbd0026eb0 migrateURI=(null) snapshot=(nil) vmop=0 flags=0x0
2020-07-21 07:57:32.953+0000: 1600130: warning : qemuDomainObjTaint:5915 : Domain id=3 name='fedora31' uuid=9e49557d-7326-4cfc-8d37-6207bb17e7f5 is tainted: host-cpu
2020-07-21 07:57:32.953+0000: 1600130: debug : qemuProcessLaunch:6745 : Building mount namespace
2020-07-21 07:57:32.953+0000: 1600130: debug : qemuProcessLaunch:6750 : Setting up raw IO
2020-07-21 07:57:32.953+0000: 1600130: debug : qemuProcessLaunch:6760 : Setting up security labelling
2020-07-21 07:57:32.965+0000: 1600130: debug : qemuProcessLaunch:6786 : QEMU vm=0x7fcbd00deae0 name=fedora31 running with pid=1618364
2020-07-21 07:57:32.965+0000: 1600130: debug : qemuProcessLaunch:6793 : Writing early domain status to disk
2020-07-21 07:57:32.966+0000: 1600130: debug : qemuProcessLaunch:6797 : Waiting for handshake from child
2020-07-21 07:57:32.968+0000: 1600130: debug : qemuProcessLaunch:6805 : Setting up domain cgroup (if required)
2020-07-21 07:57:33.015+0000: 1600130: debug : qemuSetupImagePathCgroup:74 : Allow path /var/lib/libvirt/images/disk.qcow2, perms: rw
2020-07-21 07:57:33.015+0000: 1600130: debug : qemuSetupImageCgroupInternal:134 : Not updating cgroups for disk path 'new', type: network
2020-07-21 07:57:33.016+0000: 1600130: debug : qemuProcessLaunch:6823 : Setting emulator tuning/settings
2020-07-21 07:57:33.019+0000: 1600130: debug : qemuProcessLaunch:6827 : Setting cgroup for external devices (if required)
2020-07-21 07:57:33.019+0000: 1600130: debug : qemuProcessLaunch:6831 : Setting up resctrl
2020-07-21 07:57:33.019+0000: 1600130: debug : qemuProcessLaunch:6835 : Setting up managed PR daemon
2020-07-21 07:57:33.019+0000: 1600130: debug : qemuProcessLaunch:6840 : Setting domain security labels
2020-07-21 07:57:33.028+0000: 1600130: debug : qemuProcessLaunch:6872 : Labelling done, completing handshake to child
2020-07-21 07:57:33.028+0000: 1600130: debug : qemuProcessLaunch:6875 : Handshake complete, child running
2020-07-21 07:57:33.029+0000: 1600130: debug : qemuProcessLaunch:6886 : Waiting for monitor to show up
2020-07-21 07:57:33.029+0000: 1600130: debug : qemuProcessWaitForMonitor:2393 : Connect monitor to vm=0x7fcbd00deae0 name='fedora31' retry=0
2020-07-21 07:57:33.030+0000: 1600130: info : qemuMonitorOpenInternal:717 : QEMU_MONITOR_NEW: mon=0x7fcc180430b0 fd=34
2020-07-21 07:57:33.030+0000: 1600130: debug : qemuDomainObjBeginJobInternal:830 : Starting job: job=async nested agentJob=none asyncJob=none (vm=0x7fcbd00deae0 name=fedora31, current job=none agentJob=none async=start)
2020-07-21 07:57:33.030+0000: 1600130: debug : qemuDomainObjBeginJobInternal:877 : Started job: async nested (async=start vm=0x7fcbd00deae0 name=fedora31)
2020-07-21 07:57:33.030+0000: 1600130: debug : qemuDomainObjEnterMonitorInternal:5424 : Entering monitor (mon=0x7fcc180430b0 vm=0x7fcbd00deae0 name=fedora31)
2020-07-21 07:57:33.030+0000: 1600130: debug : qemuMonitorSetCapabilities:1557 : mon:0x7fcc180430b0 vm:0x7fcbd00deae0 fd:34
2020-07-21 07:57:33.031+0000: 1600130: info : qemuMonitorSend:946 : QEMU_MONITOR_SEND_MSG: mon=0x7fcc180430b0 msg={"execute":"qmp_capabilities","id":"libvirt-1"}
 fd=-1
2020-07-21 07:57:33.174+0000: 1618375: error : qemuMonitorIORead:491 : Unable to read from monitor: Connection reset by peer
2020-07-21 07:57:33.174+0000: 1618375: debug : qemuDomainLogContextRead:6156 : Context read 0x7fcc20047000 manager=0x7fcc20035f20 inode=34977140 pos=221969
2020-07-21 07:57:33.175+0000: 1618375: error : qemuProcessReportLogError:2097 : internal error: qemu unexpectedly closed the monitor: 2020-07-21T07:57:33.126804Z qemu-kvm: -blockdev {"driver":"nbd","server":{"type":"inet","host":"XXXX","port":"10809"},"export":"new","tls-creds":"objlibvirt-1-storage_tls0","node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"}: Denied by server for option 5 (starttls)
server reported: TLS not configured
2020-07-21 07:57:33.175+0000: 1618375: debug : qemuMonitorIO:618 : Error on monitor internal error: qemu unexpectedly closed the monitor: 2020-07-21T07:57:33.126804Z qemu-kvm: -blockdev {"driver":"nbd","server":{"type":"inet","host":"XXXX","port":"10809"},"export":"new","tls-creds":"objlibvirt-1-storage_tls0","node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"}: Denied by server for option 5 (starttls)
server reported: TLS not configured
2020-07-21 07:57:33.175+0000: 1618375: debug : qemuMonitorIO:649 : Triggering error callback
2020-07-21 07:57:33.175+0000: 1618375: debug : qemuProcessHandleMonitorError:346 : Received error on 0x7fcbd00deae0 'fedora31'
2020-07-21 07:57:33.175+0000: 1600130: debug : qemuMonitorSend:958 : Send command resulted in error internal error: qemu unexpectedly closed the monitor: 2020-07-21T07:57:33.126804Z qemu-kvm: -blockdev {"driver":"nbd","server":{"type":"inet","host":"XXXX","port":"10809"},"export":"new","tls-creds":"objlibvirt-1-storage_tls0","node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"}: Denied by server for option 5 (starttls)
server reported: TLS not configured
2020-07-21 07:57:33.175+0000: 1600130: debug : qemuDomainObjExitMonitorInternal:5449 : Exited monitor (mon=0x7fcc180430b0 vm=0x7fcbd00deae0 name=fedora31)
2020-07-21 07:57:33.175+0000: 1618375: debug : qemuMonitorIO:618 : Error on monitor internal error: qemu unexpectedly closed the monitor: 2020-07-21T07:57:33.126804Z qemu-kvm: -blockdev {"driver":"nbd","server":{"type":"inet","host":"XXXX","port":"10809"},"export":"new","tls-creds":"objlibvirt-1-storage_tls0","node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"}: Denied by server for option 5 (starttls)
server reported: TLS not configured
2020-07-21 07:57:33.175+0000: 1600130: debug : qemuDomainObjEndJob:1130 : Stopping job: async nested (async=start vm=0x7fcbd00deae0 name=fedora31)
2020-07-21 07:57:33.175+0000: 1600130: debug : qemuDomainLogContextRead:6156 : Context read 0x7fcc20047000 manager=0x7fcc20035f20 inode=34977140 pos=221969
2020-07-21 07:57:33.175+0000: 1618375: debug : qemuMonitorIO:639 : Triggering EOF callback
2020-07-21 07:57:33.176+0000: 1600130: error : qemuProcessReportLogError:2097 : internal error: process exited while connecting to monitor: 2020-07-21T07:57:33.126804Z qemu-kvm: -blockdev {"driver":"nbd","server":{"type":"inet","host":"XXXX","port":"10809"},"export":"new","tls-creds":"objlibvirt-1-storage_tls0","node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"}: Denied by server for option 5 (starttls)
server reported: TLS not configured
2020-07-21 07:57:33.176+0000: 1600130: debug : qemuDomainLogContextFinalize:165 : ctxt=0x7fcc20047000
2020-07-21 07:57:33.176+0000: 1600130: debug : qemuProcessStop:7313 : Shutting down vm=0x7fcbd00deae0 name=fedora31 id=3 pid=1618364, reason=failed, asyncJob=start, flags=0x0
2020-07-21 07:57:33.176+0000: 1600130: debug : qemuDomainObjBeginJobInternal:830 : Starting job: job=async nested agentJob=none asyncJob=none (vm=0x7fcbd00deae0 name=fedora31, current job=none agentJob=none async=start)
2020-07-21 07:57:33.176+0000: 1600130: debug : qemuDomainObjBeginJobInternal:877 : Started job: async nested (async=start vm=0x7fcbd00deae0 name=fedora31)
2020-07-21 07:57:33.176+0000: 1600130: debug : qemuDomainLogAppendMessage:6224 : Append log message (vm='fedora31' message='2020-07-21 07:57:33.176+0000: shutting down, reason=failed
) stdioLogD=1
2020-07-21 07:57:33.177+0000: 1600130: info : qemuMonitorClose:866 : QEMU_MONITOR_CLOSE: mon=0x7fcc180430b0
2020-07-21 07:57:33.178+0000: 1600130: debug : qemuProcessKill:7227 : vm=0x7fcbd00deae0 name=fedora31 pid=1618364 flags=0x5
2020-07-21 07:57:33.178+0000: 1600130: debug : qemuDomainCleanupRun:6796 : driver=0x7fcbd003d710, vm=fedora31
2020-07-21 07:57:33.178+0000: 1600130: debug : qemuProcessAutoDestroyRemove:7686 : vm=fedora31
2020-07-21 07:57:33.178+0000: 1600130: warning : virSecuritySELinuxRestoreFileLabel:1461 : cannot resolve symlink new: No such file or directory
2020-07-21 07:57:33.194+0000: 1600130: debug : qemuDomainObjEndJob:1130 : Stopping job: async nested (async=start vm=0x7fcbd00deae0 name=fedora31)
2020-07-21 07:57:33.197+0000: 1600130: debug : qemuDomainObjEndAsyncJob:1168 : Stopping async job: start (vm=0x7fcbd00deae0 name=fedora31)
2020-07-21 07:57:33.197+0000: 1618375: debug : qemuProcessHandleMonitorEOF:295 : Received EOF on 0x7fcbd00deae0 'fedora31'
2020-07-21 07:57:33.197+0000: 1618375: debug : qemuMonitorDispose:225 : mon=0x7fcc180430b0
2020-07-21 07:57:33.197+0000: 1600323: debug : qemuProcessEventHandler:4898 : vm=0x7fcbd00deae0, event=7
2020-07-21 07:57:33.197+0000: 1600323: debug : qemuProcessKill:7227 : vm=0x7fcbd00deae0 name=fedora31 pid=-1 flags=0x1
2020-07-21 07:57:33.197+0000: 1600323: debug : qemuProcessKill:7231 : VM 'fedora31' not active
2020-07-21 07:57:33.197+0000: 1600323: debug : qemuDomainObjBeginJobInternal:830 : Starting job: job=destroy agentJob=none asyncJob=none (vm=0x7fcbd00deae0 name=fedora31, current job=none agentJob=none async=none)
2020-07-21 07:57:33.197+0000: 1600323: debug : qemuDomainObjBeginJobInternal:877 : Started job: destroy (async=none vm=0x7fcbd00deae0 name=fedora31)
2020-07-21 07:57:33.197+0000: 1600323: debug : processMonitorEOFEvent:4790 : Domain 0x7fcbd00deae0 'fedora31' is not active, ignoring EOF
2020-07-21 07:57:33.197+0000: 1600323: debug : qemuDomainObjEndJob:1130 : Stopping job: destroy (async=none vm=0x7fcbd00deae0 name=fedora31)
Comment 5 Peter Krempa 2020-07-27 09:12:15 UTC 
The error message is reported if the 'NBD' server doesn't offer TLS transport (it was not configured).

From your steps above, you invoke it as:

qemu-nbd --object tls-creds-x509,id=tls0,endpoint=server,dir=/etc/pki/libvirt-nbd  /tmp/test -x new -t -v

Your command line is missing the '--tls-creds=tls0' argument'.
Comment 6 Han Han 2020-07-28 06:00:51 UTC 
(In reply to Peter Krempa from comment #5)
> The error message is reported if the 'NBD' server doesn't offer TLS
> transport (it was not configured).
> 
> From your steps above, you invoke it as:
> 
> qemu-nbd --object
> tls-creds-x509,id=tls0,endpoint=server,dir=/etc/pki/libvirt-nbd  /tmp/test
> -x new -t -v
> 
> Your command line is missing the '--tls-creds=tls0' argument'.

Thank you.
It works after set '--tls-creds=tls0' in qemu-nbd.
Check with `qemu-img info`:
# qemu-img info --object secret,id=sec,data=redhat,format=raw --object tls-creds-x509,id=tls0,endpoint=client,dir=/etc/pki/libvirt-nbd/,verify-peer=yes,passwordid=sec 'json:{"file":{"driver":"nbd", "server":{"host":"XXXX", "port":10809}, "export":"new", "tls-creds":"tls0"}}'
image: nbd://XXXX:10809/new
file format: raw
virtual size: 100 MiB (104857600 bytes)
disk size: unavailable

libvirt version: v6.6.0-rc1
Comment 9 Han Han 2020-09-16 01:57:09 UTC 
Retest on libvirt-6.6.0-4.module+el8.3.0+7883+3d717aa8.x86_64 qemu-kvm-5.1.0-6.module+el8.3.0+8041+42ff16b8.x86_64, passed.
Comment 12 errata-xmlrpc 2020-11-17 17:44:45 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (virt:8.3 bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:5137