My coworker thinks it might be https://gitlab.com/libvirt/libvirt/-/blob/master/src/qemu/qemu_domain.c#L3887 which introduced the new behaviour.

> Version-Release number of selected component (if applicable):
>
> libvirt-libs-9.0.0-2.fc38.x86_64
> qemu-common-7.2.0-6.fc38.x86_64

What other 'qemu-XXXX' packages do you have. That alone doesn't provide any spice support, there are multiple spice sub-RPMs required.

qemu-ui-spice-core
qemu-char-spice
qemu-audio-spice

These are the packages on Fedora 38, we still install the same packages on Fedora 37 so I expected spice to work.

[root@fedora-38-127-0-0-2-2201 ~]# rpm -qa | grep qemu
qemu-guest-agent-7.2.0-6.fc38.x86_64
qemu-img-7.2.0-6.fc38.x86_64
qemu-user-static-x86-7.2.0-6.fc38.x86_64
qemu-user-static-arm-7.2.0-6.fc38.x86_64
qemu-user-static-aarch64-7.2.0-6.fc38.x86_64
qemu-user-static-xtensa-7.2.0-6.fc38.x86_64
qemu-user-static-sparc-7.2.0-6.fc38.x86_64
qemu-user-static-sh4-7.2.0-6.fc38.x86_64
qemu-user-static-s390x-7.2.0-6.fc38.x86_64
qemu-user-static-riscv-7.2.0-6.fc38.x86_64
qemu-user-static-ppc-7.2.0-6.fc38.x86_64
qemu-user-static-or1k-7.2.0-6.fc38.x86_64
qemu-user-static-nios2-7.2.0-6.fc38.x86_64
qemu-user-static-mips-7.2.0-6.fc38.x86_64
qemu-user-static-microblaze-7.2.0-6.fc38.x86_64
qemu-user-static-m68k-7.2.0-6.fc38.x86_64
qemu-user-static-loongarch64-7.2.0-6.fc38.x86_64
qemu-user-static-hppa-7.2.0-6.fc38.x86_64
qemu-user-static-hexagon-7.2.0-6.fc38.x86_64
qemu-user-static-cris-7.2.0-6.fc38.x86_64
qemu-user-static-alpha-7.2.0-6.fc38.x86_64
qemu-user-static-7.2.0-6.fc38.x86_64
ipxe-roms-qemu-20220210-3.git64113751.fc38.noarch
qemu-common-7.2.0-6.fc38.x86_64
qemu-ui-opengl-7.2.0-6.fc38.x86_64
qemu-system-x86-core-7.2.0-6.fc38.x86_64
qemu-ui-spice-core-7.2.0-6.fc38.x86_64
qemu-char-spice-7.2.0-6.fc38.x86_64
qemu-kvm-core-7.2.0-6.fc38.x86_64
libvirt-daemon-driver-qemu-9.0.0-2.fc38.x86_64
qemu-block-curl-7.2.0-6.fc38.x86_64
qemu-device-usb-host-7.2.0-6.fc38.x86_64
qemu-device-usb-redirect-7.2.0-6.fc38.x86_64
qemu-virtiofsd-7.2.0-6.fc38.x86_64

> error: internal error: process exited while connecting to monitor: audio: Unknown audio driver `spice'

The error is related to qemu-audio-spice, which your system seems to be lacking. Could you try installing and see if the error goes away?

Adding qemu-audio-spice makes our CI test succeed, so this is a new requirement then for spice support. Previously we could add spice graphics without requiring one to also enable audio.

Ok, so the bug here is that domcapabilities reports "spice" as a possible value for graphics, while in fact it wont work.  This is not a new bug, as far as I can tell: Having qemu-ui-spice-core installed is enough for domcapabilities to report "spice", but virt-install will fail with a internal error when qemu-char-spice is not installed as well:

# rpmquery qemu-ui-spice-core qemu-char-spice qemu-audio-spice
qemu-ui-spice-core-7.2.0-6.fc38.x86_64
package qemu-char-spice is not installed
package qemu-audio-spice is not installed

# virsh domcapabilities | grep '<graphics' -A 8
    <graphics supported='yes'>
      <enum name='type'>
        <value>sdl</value>
        <value>vnc</value>
        <value>spice</value>
        <value>egl-headless</value>
        <value>dbus</value>
      </enum>
    </graphics>

# virt-install --name subVmTest1 --os-variant cirros0.4.0 --boot hd,network --import --disk /var/tmp/cirros.qcow2 --graphics spice,listen=127.0.0.1 
Using cirros0.4.0 default --memory 256
WARNING  Unable to connect to graphical console: virt-viewer not installed. Please install the 'virt-viewer' package.
WARNING  No console to launch for the guest, defaulting to --wait -1

Starting install...
ERROR    internal error: process exited while connecting to monitor: 2023-02-17T08:39:42.597513Z qemu-system-x86_64: -chardev spicevmc,id=charchannel1,name=vdagent: 'spicevmc' is not a valid char driver name
Domain installation does not appear to have been successful.
If it was, you can restart your domain by running:
  virsh --connect qemu:///system start subVmTest1
otherwise, please restart your installation.



The cockpit-machines package recommends qemu-char-spice so that people can create Spice consoles in the UI, but Cockpit also checks the domcapabilities to figure out whether Spice will work or not.

What is the recommened way to make sure that Spice will work with libvirt?  Just installing qemu-char-spice isn't enough anymore.  How can we know which package(s) to install.

But even if that is answered, I still think that domcapabilities is not reflecting reality, and that should be fixed as well.

Just for completeness, here is what happens when domcapabilities doesn't report Spice:

# virsh domcapabilities | grep '<graphics' -A 8
    <graphics supported='yes'>
      <enum name='type'>
        <value>sdl</value>
        <value>vnc</value>
        <value>egl-headless</value>
        <value>dbus</value>
      </enum>
    </graphics>
    <video supported='yes'>

# virt-install --name subVmTest1 --os-variant cirros0.4.0 --boot hd,network --import --disk /var/tmp/cirros.qcow2 --graphics spice,listen=127.0.0.1 
Using cirros0.4.0 default --memory 256
WARNING  Unable to connect to graphical console: virt-viewer not installed. Please install the 'virt-viewer' package.
WARNING  No console to launch for the guest, defaulting to --wait -1

Starting install...
ERROR    unsupported configuration: spice graphics are not supported with this QEMU
Domain installation does not appear to have been successful.
If it was, you can restart your domain by running:
  virsh --connect qemu:///system start subVmTest1
otherwise, please restart your installation.


This is the expected behavior.  The fact that virt-install fails with an "internal error" above tells me that the bug is with libvirt and not with which packages are installed.

Thanks Marius for your investigation. In fact, bug 2064594 was very similar:"`virsh domcapabilities` claims spice graphics support without QXL driver". Adjusting the title a bit.

Similar reproducer with slightly different error message, which matches the error message in the original description:

rpm -qa|grep spice
# spice-server-0.15.1-2.fc38.x86_64
# qemu-ui-spice-core-7.2.0-6.fc38.x86_64
# qemu-char-spice-7.2.0-6.fc38.x86_64

   → This is a valid setup; if virt-install or libvirt assume more packages (like qemu-audio-spice), that should be expressed via package dependencies, but then, why even bother splitting out the package?

curl -L -o /tmp/cirros.qcow2 https://download.cirros-cloud.net/0.4.0/cirros-0.4.0-i386-disk.img
echo '#cloud-config' > /tmp/userdata

virt-install --name c --os-variant cirros0.4.0 --memory 128 --wait -1 --noautoconsole --disk 'size=0.1,format=qcow2,backing_store=/tmp/cirros.qcow2' --graphics 'spice,listen=127.0.0.1'  --cloud-init 'user-data=/tmp/userdata'

This fails with

Starting install...
Allocating 'c.qcow2'                                                                                             |    0 B  00:00:00 ... 
Removing disk 'c.qcow2'                                                                                          |    0 B  00:00:00     
ERROR    internal error: process exited while connecting to monitor: audio: Unknown audio driver `spice'
Domain installation does not appear to have been successful.

Some more investigations:

 - When removing all *spice* packages (removing qemu-char-spice takes everything else with it, like qemu-ui-spice-core and spice-server), then `virsh domcapabilities` correctly reports no "spice" capability.
 - Installing qemu-char-spice alone already depends on qemu-ui-spice-core and spice-server, so the packages really can't be installed/uninstalled independently from each other. In other words, splitting them makes little sense. Now `domcapabilities` claims spice, and the bug happens.

So either qemu-ui-spice-core should grow a `Requires: qemu-audio-spice` (it's just an extra 20 kB, and splitting it out is not worth it IMHO), or better, virt-install/qemu/etc. should be fixed to not require spice audio when requesting spice video.

I'd like to work around this in cockpit-machines. For 2064594 I called `qemu-system-x86_64 -vga help` and checked for `qxl` presence. But there is no corresponding option for audio drivers. So I suppose I'll just add

   %if 0%{?fedora} >= 38
   Requires: (qemu-audio-spice if qemu-ui-spice-core)
   %endif

to cockpit-machines RPM as a hack until this gets fixed.

Thanks!

Any chance that this can be looked at soon? Fedora 38 is released now. Adding the dependency to qemu-ui-spice-core is an easy bandaid (if you think that should be done, please reassign to qemu). Fixing the detection would be better obviously, but more complicated, and saving 20 kB may just simply not be worth it.

libvirt is the one unconditionally adding the spice audio device, when spice graphics is requested in the domain XML. So libvirt has the bug here. I filed an issue upstream: https://gitlab.com/libvirt/libvirt/-/issues/473

In the meantime I'll look at tweaking qemu packaging, so moving this to qemu

This message is a reminder that Fedora Linux 38 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora Linux 38 on 2024-05-21.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
'version' of '38'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, change the 'version' 
to a later Fedora Linux version. Note that the version field may be hidden.
Click the "Show advanced fields" button if you do not see it.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora Linux 38 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora Linux, you are encouraged to change the 'version' to a later version
prior to this bug being closed.

Confirmed on current Fedora 40:

# dnf remove qemu-audio-spice
# virsh domcapabilities | grep '<graphics' -A 8
    <graphics supported='yes'>
      <enum name='type'>
        <value>sdl</value>
        <value>vnc</value>
        <value>spice</value>         ← claims spice support
        <value>egl-headless</value>
        <value>dbus</value>
      </enum>
    </graphics>


# virt-install --memory 50 --pxe --virt-type qemu --os-variant alpinelinux3.8 --disk none --graphics spice,listen=127.0.0.1 --wait 0 --name test1
ERROR    internal error: process exited while connecting to monitor: 2024-05-08T15:21:02.312725Z qemu-system-x86_64: Unknown audio driver `spice'

This message is a reminder that Fedora Linux 40 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora Linux 40 on 2025-05-13.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
'version' of '40'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, change the 'version' 
to a later Fedora Linux version. Note that the version field may be hidden.
Click the "Show advanced fields" button if you do not see it.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora Linux 40 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora Linux, you are encouraged to change the 'version' to a later version
prior to this bug being closed.

Confirmed on current Fedora 42 with reproducer from comment #14