Bug 1991671

Summary: vmstate differs between -audiodev and QEMU_AUDIO_DRV when no sound frontends devs present.
Product: Red Hat Enterprise Linux Advanced Virtualization Reporter: Daniel Berrangé <berrange>
Component: qemu-kvmAssignee: Dr. David Alan Gilbert <dgilbert>
qemu-kvm sub component: Audio QA Contact: Guo, Zhiyi <zhguo>
Status: CLOSED ERRATA Docs Contact:
Severity: high    
Priority: high CC: coli, ddepaula, dgilbert, jinzhao, juzhang, kkiwi, kraxel, lizhu, virt-maint, zhguo
Version: 8.5Keywords: Triaged
Target Milestone: rc   
Target Release: 8.5   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: qemu-kvm-6.0.0-28.module+el8.5.0+12271+fffa967b Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1992197 (view as bug list) Environment:
Last Closed: 2021-11-16 07:55:01 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1957194, 1977891, 1992197    

Description Daniel Berrangé 2021-08-09 16:29:35 UTC
Description of problem:

Consider the two ways of configuring the QEMU audio backend

$ QEMU_AUDIO_DRV=none ./qemu-system-x86_64 -cdrom ~/Downloads/Fedora-Workstation-Live-x86_64-32-1.6.iso -S -monitor stdio

and

$ qemu-system-x86_64 -cdrom ~/Downloads/Fedora-Workstation-Live-x86_64-32-1.6.iso -S -monitor stdio -audiodev none,id=a

These both result in the exact same guest ABI, since nothing about audio backend affects the guest. They have the exact same backend behaviour too.

When live migrating, however, their vmstate differs

The config using -audiodev contains an extra "audio" section in its vmstate.

This appears to be because -audiodev is processed unconditionally at QEMU startup, while $QEMU_AUDIO_DRV is processed lazily when the first sound device is added to the VM (whether at cold boot, or hotplug at runtime).

Due to the lazy init for $QEMU_AUDIO_DRV, the 'audio' section doesn't get added to vmstate.

Amuzingly/frustratingly the "audio" vmstate section is entirely empty, so it serves no functional purpose during live migration, except to break compatibility in this way.


This has unfortunately broken live migration backwards compatibility since libvirt switched to using -audiodev instead of $QEMU_AUDIO_DRV, since libvirt will configure an audio backend regardless of whether any sound frontend device exists.

Version-Release number of selected component (if applicable):
qemu-kvm-6.0.0-26.module+el8.5.0+12044+525f0ebc.x86_64

How reproducible:
Always

Steps to Reproduce:
1. $ QEMU_AUDIO_DRV=none ./qemu-system-x86_64 -cdrom ~/Downloads/Fedora-Workstation-Live-x86_64-32-1.6.iso -S -monitor stdio
2. migrate 'exec:dd of=env.img'
3. $ qemu-system-x86_64 -cdrom ~/Downloads/Fedora-Workstation-Live-x86_64-32-1.6.iso -S -monitor stdio -audiodev none,id=a
4. migrate 'exec:dd of=args.img'
5. scripts/analyze-migration.py -f env.img
6. scripts/analyze-migration.py -f args.img

Actual results:
Extra "audio" section in args.img

Expected results:
vmstate is identical for both ways of configuring audio backend.

Additional info:

Comment 1 Dr. David Alan Gilbert 2021-08-09 16:34:37 UTC
I think we might be able to fix this just by making a .needed on the audio vmstate to say never to send it.
Old qemu's should ignore the absence of one.

Comment 2 Dr. David Alan Gilbert 2021-08-09 17:40:18 UTC
Posted upstream as:

audio: Never send migration section

now, which downstream qemu's do you want that in?

Comment 3 Daniel Berrangé 2021-08-10 12:42:21 UTC
(In reply to Dr. David Alan Gilbert from comment #2)
> Posted upstream as:
> 
> audio: Never send migration section
> 
> now, which downstream qemu's do you want that in?

RHEL-AV 8.5 and RHEL-9.0  will both have libvirt versions which trigger this bug, so QEMU in those two streams.  Base RHEL 8.5 will still have older libvirt so doesn't need this in base QEMU.

Comment 4 Dr. David Alan Gilbert 2021-08-10 17:52:50 UTC
OK, created 1992197 as a 9 clone; although 9 final will pick it up anyway through rebase, so we might not have to do much.

Comment 5 Dr. David Alan Gilbert 2021-08-10 18:33:31 UTC
Fix went in upstream as da77adbaf619c4d170cb

Comment 7 Danilo de Paula 2021-08-11 13:53:04 UTC
QA ACK, please?

Comment 8 Guo, Zhiyi 2021-08-12 01:18:31 UTC
(In reply to Danilo Cesar Lemes de Paula from comment #7)
> QA ACK, please?

Done!

Comment 13 Yanan Fu 2021-08-19 06:35:22 UTC
QE bot(pre verify): Set 'Verified:Tested,SanityOnly' as gating/tier1 test pass.

Comment 14 Guo, Zhiyi 2021-08-19 08:01:21 UTC
Check this bug from libvirt layer:

pkgs on rhel8.4 host:
qemu-kvm-5.2.0-16.module+el8.4.0+10806+b7d97207.x86_64
libvirt-7.0.0-14.1.module+el8.4.0+11095+d46acebf.x86_64

pkgs on rhel8.5 host:
qemu-kvm-6.0.0-28.module+el8.5.0+12271+fffa967b.x86_64
libvirt-7.6.0-2.module+el8.5.0+12219+a5ea13d2.x86_64

Steps:
Boot a VM with qemu cli generated by libvirt on rhel 8.4 host:
# cat /var/log/libvirt/qemu/rhel84.log
2021-05-24-02:28:14, ), qemu version: 5.2.0qemu-kvm-5.2.0-16.module+el8.4.0+10806+b7d97207, kernel: 4.18.0-305.el8.x86_64, hostname: smicro-x12s-01.rhts.eng.pek2.re
dhat.com
LC_ALL=C \
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin \
HOME=/var/lib/libvirt/qemu/domain-1-rhel84 \
XDG_DATA_HOME=/var/lib/libvirt/qemu/domain-1-rhel84/.local/share \
XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain-1-rhel84/.cache \
XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain-1-rhel84/.config \
QEMU_AUDIO_DRV=none \
/usr/libexec/qemu-kvm \
-name guest=rhel84,debug-threads=on \
...
-machine pc-q35-rhel8.4.0,accel=kvm,usb=off,dump-guest-core=off,memory-backend=pc.ram \
-cpu Skylake-Client,ss=on,vmx=on,pdcm=on,hypervisor=on,tsc-adjust=on,mpx=on,umip=on,md-clear=on,xsaves=on,ibpb=on,ibrs=on,amd-stibp=on,amd-ssbd=on,hle=off,rtm=off,a
vx512f=off,avx512dq=off,clwb=off,avx512cd=off,avx512bw=off,avx512vl=off,avx512vnni=off,avx512-bf16=off,taa-no=off \
...
-vnc 0.0.0.0:0 \
-device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vram64_size_mb=0,vgamem_mb=16,max_outputs=1,bus=pcie.0,addr=0x1 \
...

Then migrate this VM to rhel 8.5 host, check the qemu cli generated, it becomes:
2021-08-19 07:42:33.358+0000: starting up libvirt version: 7.6.0, package: 2.module+el8.5.0+12219+a5ea13d2 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>, 202
1-08-12-05:59:49, ), qemu version: 6.0.0qemu-kvm-6.0.0-28.module+el8.5.0+12271+fffa967b, kernel: 4.18.0-330.el8.x86_64, hostname: smicro-x12s-02.rhts.eng.pek2.redha
t.com
LC_ALL=C \
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin \
HOME=/var/lib/libvirt/qemu/domain-1-rhel84 \
XDG_DATA_HOME=/var/lib/libvirt/qemu/domain-1-rhel84/.local/share \
XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain-1-rhel84/.cache \
XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain-1-rhel84/.config \
/usr/libexec/qemu-kvm \
-name guest=rhel84,debug-threads=on \
-S \
-object '{"qom-type":"secret","id":"masterKey0","format":"raw","file":"/var/lib/libvirt/qemu/domain-1-rhel84/master-key.aes"}' \
-machine pc-q35-rhel8.4.0,accel=kvm,usb=off,dump-guest-core=off,memory-backend=pc.ram \
-cpu Skylake-Client,ss=on,vmx=on,pdcm=on,hypervisor=on,tsc-adjust=on,mpx=on,umip=on,md-clear=on,xsaves=on,ibpb=on,ibrs=on,amd-stibp=on,amd-ssbd=on,hle=off,rtm=off,avx512f=off,avx512dq=off,clwb=off,avx512cd=off,avx512bw=off,avx512vl=off,avx512vnni=off,avx512-bf16=off,taa-no=off \
...
-audiodev id=audio1,driver=none \
-vnc 0.0.0.0:0,audiodev=audio1 \
-device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vram64_size_mb=0,vgamem_mb=16,max_outputs=1,bus=pcie.0,addr=0x1 \
...

Then migrate this VM back, now the qemu cli has changed back to original with QEMU_AUDIO_DRV=none

Comment 15 Guo, Zhiyi 2021-08-19 08:20:39 UTC
Mark verified per comment 14 as ping-pong migration between 8.4 & 8.5 doesn't have audiodev/QEMU_AUDIO_DRV on the fix build qemu-kvm-6.0.0-28.module+el8.5.0+12271+fffa967b.x86_64.

Comment 17 errata-xmlrpc 2021-11-16 07:55:01 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (virt:av bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:4684