Bug 1903137

Summary: nbd+tls: Boot guest very slowly compared to boot with nbd image
Product: Red Hat Enterprise Linux 9 Reporter: zixchen
Component: qemu-kvmAssignee: Eric Blake <eblake>
qemu-kvm sub component: NBD QA Contact: Tingting Mao <timao>
Status: CLOSED DUPLICATE Docs Contact:
Severity: medium    
Priority: low CC: coli, virt-maint, xuwei
Version: 9.0Keywords: Triaged
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-03-07 08:30:20 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
boot log with nbd+tls
none
boot log none

Description zixchen 2020-12-01 12:26:19 UTC
Description of problem:
Boot a guest with nbd tls is very slowly, over 5 to 7 minutes, compares to boot with nbd image only takes less than one minute on the same host.

Version-Release number of selected component (if applicable):
qemu-kvm-5.2.0-0.module+el8.4.0+8855+a9e237a9.x86_64
kernel-4.18.0-255.el8.x86_64

How reproducible:
100%

Steps to Reproduce:
1.Boot a guest with nbd tls:
   /usr/libexec/qemu-kvm \
    -S  \
    -name &apos;avocado-vt-vm1&apos;  \
    -sandbox on  \
    -machine q35 \
    -device pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x1,chassis=1 \
    -device pcie-pci-bridge,id=pcie-pci-bridge-0,addr=0x0,bus=pcie-root-port-0  \
    -nodefaults \
    -device VGA,bus=pcie.0,addr=0x2 \
    -m 15360  \
    -smp 16,maxcpus=16,cores=8,threads=1,dies=1,sockets=2  \
    -cpu &apos;IvyBridge&apos;,+kvm_pv_unhalt \
    -chardev socket,nowait,path=/var/tmp/avocado_7dtfvkdp/monitor-qmpmonitor1-20201201-002128-ibfqK4j4,id=qmp_id_qmpmonitor1,server  \
    -mon chardev=qmp_id_qmpmonitor1,mode=control \
    -chardev socket,nowait,path=/var/tmp/avocado_7dtfvkdp/monitor-catch_monitor-20201201-002128-ibfqK4j4,id=qmp_id_catch_monitor,server  \
    -mon chardev=qmp_id_catch_monitor,mode=control \
    -device pvpanic,ioport=0x505,id=idzxrwEb \
    -chardev socket,nowait,path=/var/tmp/avocado_7dtfvkdp/serial-serial0-20201201-002128-ibfqK4j4,id=chardev_serial0,server \
    -device isa-serial,id=serial0,chardev=chardev_serial0  \
    -chardev socket,id=seabioslog_id_20201201-002128-ibfqK4j4,path=/var/tmp/avocado_7dtfvkdp/seabios-20201201-002128-ibfqK4j4,server,nowait \
    -device isa-debugcon,chardev=seabioslog_id_20201201-002128-ibfqK4j4,iobase=0x402 \
    -device pcie-root-port,id=pcie-root-port-1,port=0x1,addr=0x1.0x1,bus=pcie.0,chassis=2 \
    -device qemu-xhci,id=usb1,bus=pcie-root-port-1,addr=0x0 \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \
    -object tls-creds-x509,id=image1_access,endpoint=client,dir=/etc/pki/qemu-kvm \
    -device pcie-root-port,id=pcie-root-port-2,port=0x2,addr=0x1.0x2,bus=pcie.0,chassis=3 \
    -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie-root-port-2,addr=0x0 \
    -blockdev node-name=nbd_image1,driver=nbd,auto-read-only=on,discard=unmap,server.type=inet,server.host=hp-dl388pg8-01.lab.eng.pek2.redhat.com,server.port=10809,tls-creds=image1_access,cache.direct=on,cache.no-flush=off \
    -blockdev node-name=drive_image1,driver=raw,read-only=off,cache.direct=on,cache.no-flush=off,file=nbd_image1 \
    -device scsi-hd,id=image1,drive=drive_image1,write-cache=on \
    -device pcie-root-port,id=pcie-root-port-3,port=0x3,addr=0x1.0x3,bus=pcie.0,chassis=4 \
    -device virtio-net-pci,mac=9a:81:39:e1:38:45,id=idRlYHRN,netdev=idxQUDeq,bus=pcie-root-port-3,addr=0x0  \
    -netdev tap,id=idxQUDeq,vhost=on,vhostfd=20,fd=7 \
    -vnc :0  \
    -rtc base=utc,clock=host,driftfix=slew  \
    -boot menu=off,order=cdn,once=d,strict=off  \
    -no-shutdown \
    -enable-kvm \
2. guest boots successfully after 5-7 minutes. Attach with a boot log. Specially, guest hangs for around 3 minutes just after bootloader selects to boot from rhel8.4 and before it gets the first boot log. 
3. reboot 5 times.



Actual results:
Boot very slowly, sometimes over 7 mins, but no shorter than 5 mins. However, on the same host, 
boot a VM with nbd disk image only takes less than one minute.

Expected results:
Boot a guest should take a shorter time.


Additional info:
run io, looks normal.
# dd if=/dev/sda of=/dev/zero bs=4k count=100k
102400+0 records in
102400+0 records out
419430400 bytes (419 MB, 400 MiB) copied, 1.30025 s, 323 MB/s

Comment 1 zixchen 2020-12-01 12:30:13 UTC
Created attachment 1735212 [details]
boot log with nbd+tls

Comment 2 zixchen 2020-12-01 12:33:50 UTC
Created attachment 1735213 [details]
boot log

Comment 3 John Ferlan 2021-09-08 21:46:55 UTC
Move RHEL-AV bugs to RHEL9. If necessary to resolve in RHEL8, then clone to the current RHEL8 release.

Comment 4 Tingting Mao 2022-03-07 08:30:20 UTC

*** This bug has been marked as a duplicate of bug 1977582 ***