Bug 2156300

I can reproduce this on s390x. It seems the first time libvirt tries to remove 'console0' instead of 'serial0':

# virsh dumpxml avocado-vt-vm1 --xpath //console
<console type="pty" tty="/dev/pts/2">
  <source path="/dev/pts/2"/>
  <target type="serial" port="0"/>
  <alias name="serial0"/>
</console>
<console type="pty">
  <source path="/dev/pts/3"/>
  <target type="virtio" port="1"/>
  <alias name="console1"/>
</console>

journal:

QEMU_MONITOR_SEND_MSG: mon=0x3ffa804a2f0 msg={"execute":"device_del","arguments":{"id":"console0"},"id":"libvirt-14"}
Line [{"id": "libvirt-14", "error": {"class": "DeviceNotFound", "desc": "Device 'console0' not found"}}]


However after the virtqemud restart:

QEMU_MONITOR_SEND_MSG: mon=0x3ff501f9010 msg={"execute":"device_del","arguments":{"id":"serial0"},"id":"libvirt-12"}
QEMU_MONITOR_RECV_REPLY: mon=0x3ff501f9010 reply={"id": "libvirt-12", "error": {"class": "GenericError", "desc": "Bus 's390-sclp-events-bus.0' does not support hotplugging"}}

Patch posted on the list:

https://listman.redhat.com/archives/libvir-list/2023-March/238763.html

Merged upstream as:

commit 42d53ac799a1d7f1414737caa4deb73871876992
Author:     Michal Prívozník <mprivozn>
AuthorDate: Tue Mar 14 11:03:50 2023 +0100
Commit:     Michal Prívozník <mprivozn>
CommitDate: Wed Mar 15 12:35:27 2023 +0100

    qemu_alias: Fix backcompat console alias generation
    
    We have this crazy backwards compatibility when it comes to
    serial and console devices. Basically, in same cases the very
    first <console/> is just an alias to the very first <serial/>
    device. This is to be seen at various places:
    
    1) virDomainDefFormatInternalSetRootName() - when generating
       domain XML, the <console/> configuration is basically ignored
       and corresponding <serial/> config is formatted,
    
    2) virDomainDefAddConsoleCompat() - which adds a copy of
       <serial/> or <console/> into virDomainDef in post parse.
    
    And when talking to QEMU we need a special handling too, because
    while <serial/> is generated on the cmd line, the <console/> is
    not. And in a lot of place we get it right. Except for generating
    device aliases. On domain startup the 'expected' happens and
    devices get "serial0" and "console0" aliases, correspondingly.
    This ends up in the status XML too. But due to aforementioned
    trick when formatting domain XML, "serial0" ends up in both
    'virsh dumpxml' and the status XML. But internally, both devices
    have different alias. Therefore, detaching the device using
    <console/> fails as qemuDomainDetachDeviceChr() tries to detach
    "console0".
    
    After the daemon is restarted and status XML is parsed, then
    everything works suddenly. This is because in the status XML both
    devices have the same alias.
    
    Let's generate correct alias from the beginning.
    
    Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=2156300
    Signed-off-by: Michal Privoznik <mprivozn>
    Reviewed-by: Andrea Bolognani <abologna>

v9.1.0-206-g42d53ac799

I have tested on libvirt-9.2.0-1.el9.x86_64
Scenario 1:
1. start a vm with tty console
# virsh dumpxml rhel9 --xpath //console
<console type="pty" tty="/dev/pts/1">
  <source path="/dev/pts/1"/>
  <target type="serial" port="0"/>
  <alias name="serial0"/>
</console>

2. try to detach the console device
# cat console.xml 
<console type="pty" tty="/dev/pts/1">
  <source path="/dev/pts/1"/>
  <target type="serial" port="0"/>
  <alias name="serial0"/>
</console>

# virsh detach-device rhel9 console.xml 
error: Failed to detach device from console.xml
error: device not found: chr type 'console' device not present in domain configuration

...............I think this error message does not make sense.

Scenario 2:
edit the serial device, change it to pty device then try unplug the console device

[root@zhetang-regression-test images]# virsh detach-device rhel9 console.xml 
Device detached successfully

[root@zhetang-regression-test images]# virsh dumpxml rhel9
error: internal error: unexpected char device type -1742048048

[root@zhetang-regression-test images]# virsh console rhel9
Connected to domain 'rhel9'
Escape character is ^] (Ctrl + ])
error: operation failed: PTY device is not yet assigned

[root@zhetang-regression-test images]# cat /tmp/virtqemud 
2023-04-10 03:45:11.931+0000: 49995: info : libvirt version: 9.2.0, package: 1.el9 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>, 2023-04-03-11:47:57, )
2023-04-10 03:45:11.931+0000: 49995: info : hostname: zhetang-regression-test
2023-04-10 03:45:11.931+0000: 49995: error : qemuMonitorJSONCheckErrorFull:368 : internal error: unable to execute QEMU command 'chardev-remove': Chardev 'charserial0' not found
2023-04-10 03:46:31.607+0000: 49976: error : virDomainChrDefFormat:24407 : internal error: unexpected char device type 1486489991
2023-04-10 03:46:31.607+0000: 49976: warning : qemuDomainSaveStatus:6259 : Failed to save status on vm rhel9
2023-04-10 03:46:39.589+0000: 49978: error : virDomainChrDefFormat:24407 : internal error: unexpected char device type 1486489991

(In reply to zhentang from comment #4)
> I have tested on libvirt-9.2.0-1.el9.x86_64
> Scenario 1:
> 1. start a vm with tty console
> # virsh dumpxml rhel9 --xpath //console
> <console type="pty" tty="/dev/pts/1">
>   <source path="/dev/pts/1"/>
>   <target type="serial" port="0"/>
>   <alias name="serial0"/>
> </console>
> 
> 2. try to detach the console device
> # cat console.xml 
> <console type="pty" tty="/dev/pts/1">
>   <source path="/dev/pts/1"/>
>   <target type="serial" port="0"/>
>   <alias name="serial0"/>
> </console>
> 
> # virsh detach-device rhel9 console.xml 
> error: Failed to detach device from console.xml
> error: device not found: chr type 'console' device not present in domain
> configuration
> 

Alright, this one I can reproduce, but ...

> ...............I think this error message does not make sense.
> 
> Scenario 2:
> edit the serial device, change it to pty device then try unplug the console
> device
> 

What do you mean by this exactly? The serial device is already type of "pty".

> [root@zhetang-regression-test images]# virsh detach-device rhel9 console.xml 
> Device detached successfully
> 
> [root@zhetang-regression-test images]# virsh dumpxml rhel9
> error: internal error: unexpected char device type -1742048048

OUCH, this shouldn't happen! This means we screw up big time.

(In reply to Michal Privoznik from comment #5)
> (In reply to zhentang from comment #4)
> > I have tested on libvirt-9.2.0-1.el9.x86_64
> > Scenario 1:
> > 1. start a vm with tty console
> > # virsh dumpxml rhel9 --xpath //console
> > <console type="pty" tty="/dev/pts/1">
> >   <source path="/dev/pts/1"/>
> >   <target type="serial" port="0"/>
> >   <alias name="serial0"/>
> > </console>
> > 
> > 2. try to detach the console device
> > # cat console.xml 
> > <console type="pty" tty="/dev/pts/1">
> >   <source path="/dev/pts/1"/>
> >   <target type="serial" port="0"/>
> >   <alias name="serial0"/>
> > </console>
> > 
> > # virsh detach-device rhel9 console.xml 
> > error: Failed to detach device from console.xml
> > error: device not found: chr type 'console' device not present in domain
> > configuration
> > 
> 
> Alright, this one I can reproduce, but ...
> 
> > ...............I think this error message does not make sense.
> > 
> > Scenario 2:
> > edit the serial device, change it to pty device then try unplug the console
> > device
> > 
> 
> What do you mean by this exactly? The serial device is already type of "pty".
Sorry, that's a typo. What I want to say is change to "pci-serial"

> 
> > [root@zhetang-regression-test images]# virsh detach-device rhel9 console.xml 
> > Device detached successfully
> > 
> > [root@zhetang-regression-test images]# virsh dumpxml rhel9
> > error: internal error: unexpected char device type -1742048048
> 
> OUCH, this shouldn't happen! This means we screw up big time.

(In reply to zhentang from comment #6)
> > What do you mean by this exactly? The serial device is already type of "pty".
> Sorry, that's a typo. What I want to say is change to "pci-serial"
> 

I'm still unable to reproduce. Do you mean <serial type='dev'> <target> <model name='pci-serial'/> </target> </serial>? Please paste the full XML. The more information the better.

(In reply to Michal Privoznik from comment #7)
> (In reply to zhentang from comment #6)
> > > What do you mean by this exactly? The serial device is already type of "pty".
> > Sorry, that's a typo. What I want to say is change to "pci-serial"
> > 
> 
> I'm still unable to reproduce. Do you mean <serial type='dev'> <target>
> <model name='pci-serial'/> </target> </serial>? Please paste the full XML.
> The more information the better.

ok, I have made another try and this is the full xml. Seems like this error would occurs when a pci-serial serial device and a console device exists,then detach the console device.
 

[root@zhetang-regression-test images]# virsh dumpxml rhel9
<domain type='kvm' id='4'>
  <name>rhel9</name>
  <uuid>2b7dbda9-6025-4e4c-80ee-b865e2a02013</uuid>
  <metadata>
    <libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
      <libosinfo:os id="http://redhat.com/rhel/9-unknown"/>
    </libosinfo:libosinfo>
  </metadata>
  <memory unit='KiB'>3072000</memory>
  <currentMemory unit='KiB'>3072000</currentMemory>
  <vcpu placement='static'>2</vcpu>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64' machine='pc-q35-rhel9.2.0'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <vmport state='off'/>
  </features>
  <cpu mode='custom' match='exact' check='full'>
    <model fallback='forbid'>Skylake-Server-IBRS</model>
    <vendor>Intel</vendor>
    <feature policy='require' name='ss'/>
    <feature policy='require' name='vmx'/>
    <feature policy='require' name='hypervisor'/>
    <feature policy='require' name='tsc_adjust'/>
    <feature policy='require' name='clflushopt'/>
    <feature policy='require' name='umip'/>
    <feature policy='require' name='pku'/>
    <feature policy='require' name='md-clear'/>
    <feature policy='require' name='stibp'/>
    <feature policy='require' name='arch-capabilities'/>
    <feature policy='require' name='ssbd'/>
    <feature policy='require' name='ibpb'/>
    <feature policy='require' name='ibrs'/>
    <feature policy='require' name='amd-stibp'/>
    <feature policy='require' name='amd-ssbd'/>
    <feature policy='require' name='rsba'/>
    <feature policy='require' name='skip-l1dfl-vmentry'/>
    <feature policy='require' name='pschange-mc-no'/>
    <feature policy='disable' name='mpx'/>
  </cpu>
  <clock offset='utc'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/RHEL-9.3.0-20230325.0-x86_64.qcow2' index='1'/>
      <backingStore/>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </disk>
    <controller type='usb' index='0' model='qemu-xhci' ports='15'>
      <alias name='usb'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
    </controller>
    <controller type='sata' index='0'>
      <alias name='ide'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'>
      <alias name='pcie.0'/>
    </controller>
    <controller type='pci' index='1' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='1' port='0x10'/>
      <alias name='pci.1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='2' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='2' port='0x11'/>
      <alias name='pci.2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='3' port='0x12'/>
      <alias name='pci.3'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/>
    </controller>
    <controller type='pci' index='4' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='4' port='0x13'/>
      <alias name='pci.4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/>
    </controller>
    <controller type='pci' index='5' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='5' port='0x14'/>
      <alias name='pci.5'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x4'/>
    </controller>
    <controller type='pci' index='6' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='6' port='0x15'/>
      <alias name='pci.6'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x5'/>
    </controller>
    <controller type='pci' index='7' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='7' port='0x16'/>
      <alias name='pci.7'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x6'/>
    </controller>
    <controller type='pci' index='8' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='8' port='0x17'/>
      <alias name='pci.8'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x7'/>
    </controller>
    <controller type='pci' index='9' model='pcie-to-pci-bridge'>
      <model name='pcie-pci-bridge'/>
      <alias name='pci.9'/>
      <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <alias name='virtio-serial0'/>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </controller>
    <interface type='network'>
      <mac address='52:54:00:d4:17:42'/>
      <source network='default' portid='150d1e5c-d02f-4ba4-b924-1f29efd355c0' bridge='virbr0'/>
      <target dev='vnet3'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </interface>
    <serial type='pty'>
      <source path='/dev/pts/3'/>
      <target type='pci-serial' port='0'>
        <model name='pci-serial'/>
      </target>
      <alias name='serial0'/>
      <address type='pci' domain='0x0000' bus='0x09' slot='0x01' function='0x0'/>
    </serial>
    <console type='pty' tty='/dev/pts/3'>
      <source path='/dev/pts/3'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
      <address type='pci' domain='0x0000' bus='0x09' slot='0x01' function='0x0'/>
    </console>
    <channel type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/domain-4-rhel9/org.qemu.guest_agent.0'/>
      <target type='virtio' name='org.qemu.guest_agent.0' state='disconnected'/>
      <alias name='channel0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='tablet' bus='usb'>
      <alias name='input0'/>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'>
      <alias name='input1'/>
    </input>
    <input type='keyboard' bus='ps2'>
      <alias name='input2'/>
    </input>
    <graphics type='vnc' port='5900' autoport='yes' listen='127.0.0.1'>
      <listen type='address' address='127.0.0.1'/>
    </graphics>
    <sound model='ich9'>
      <alias name='sound0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1b' function='0x0'/>
    </sound>
    <audio id='1' type='none'/>
    <video>
      <model type='virtio' heads='1' primary='yes'/>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
    </video>
    <watchdog model='itco' action='reset'>
      <alias name='watchdog0'/>
    </watchdog>
    <memballoon model='virtio'>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </memballoon>
    <rng model='virtio'>
      <backend model='random'>/dev/urandom</backend>
      <alias name='rng0'/>
      <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
    </rng>
  </devices>
  <seclabel type='dynamic' model='selinux' relabel='yes'>
    <label>system_u:system_r:svirt_t:s0:c83,c893</label>
    <imagelabel>system_u:object_r:svirt_image_t:s0:c83,c893</imagelabel>
  </seclabel>
  <seclabel type='dynamic' model='dac' relabel='yes'>
    <label>+107:+107</label>
    <imagelabel>+107:+107</imagelabel>
  </seclabel>
</domain>


[root@zhetang-regression-test images]# cat console.xml 
<console type='pty' tty='/dev/pts/3'>
      <source path='/dev/pts/3'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>

[root@zhetang-regression-test images]# virsh detach-device rhel9 console.xml
Device detached successfully

[root@zhetang-regression-test images]# virsh dumpxml rhel9
error: internal error: unexpected char device type -1073741280

[root@zhetang-regression-test images]# virsh list
 Id   Name    State
-----------------------
 4    rhel9   running

[root@zhetang-regression-test images]# virsh destroy rhel9
error: Disconnected from qemu:///system due to end of file
error: Failed to destroy domain 'rhel9'
error: End of file while reading data: Input/output error


virtqemud log
2023-04-11 13:28:14.899+0000: 50986: error : virDomainChrDefFormat:24407 : internal error: unexpected char device type -1069478160
2023-04-11 13:28:14.899+0000: 50986: warning : qemuDomainSaveStatus:6259 : Failed to save status on vm rhel9
2023-04-11 13:28:26.499+0000: 50982: error : virDomainChrDefFormat:24407 : internal error: unexpected char device type -1073741280
2023-04-11 13:31:14.232+0000: 50984: error : qemuDomainOpenConsole:13989 : internal error: character device rhel9 is not using a PTY
2023-04-11 13:31:29.074+0000: 50986: error : virDomainChrDefFormat:24407 : internal error: unexpected char device type -1073713264
2023-04-11 13:39:55.449+0000: 50984: error : virDomainChrDefFormat:24407 : internal error: unexpected char device type -1073713264
2023-04-11 13:39:55.449+0000: 50984: warning : qemuDomainSaveStatus:6259 : Failed to save status on vm rhel9
2023-04-11 13:39:55.451+0000: 51922: error : virDomainChrDefFormat:24407 : internal error: unexpected char device type -1073713264
2023-04-11 13:39:55.451+0000: 51922: warning : qemuDomainSaveStatus:6259 : Failed to save status on vm rhel9
2023-04-11 13:39:55.943+0000: 52102: info : libvirt version: 9.2.0, package: 1.el9 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>, 2023-04-03-11:47:57, )
2023-04-11 13:39:55.943+0000: 52102: info : hostname: zhetang-regression-test
2023-04-11 13:39:55.943+0000: 52102: error : qemuDomainMasterKeyReadFile:614 : internal error: domain master key file doesn't exist in /var/lib/libvirt/qemu/domain-4-rhel9

===============================================================================================================
And here is another issue found. 
when define a pci-serial device and start the guest, it would not add a console device.  
[root@zhetang-regression-test images]# virsh dumpxml  rhel9 | grep -A 10 "serial type"
    <serial type='pty'>
      <source path='/dev/pts/0'/>
      <target type='pci-serial' port='0'>
        <model name='pci-serial'/>
      </target>
      <alias name='serial0'/>
      <address type='pci' domain='0x0000' bus='0x09' slot='0x01' function='0x0'/>
    </serial>
    <channel type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/domain-5-rhel9/org.qemu.guest_agent.0'/>
      <target type='virtio' name='org.qemu.guest_agent.0' state='connected'/>


But we could also start the guest with a console device represent the same device with the serial device

<serial type='pty'>
      <source path='/dev/pts/3'/>
      <target type='pci-serial' port='0'>
        <model name='pci-serial'/>
      </target>
      <alias name='serial0'/>
      <address type='pci' domain='0x0000' bus='0x09' slot='0x01' function='0x0'/>
    </serial>
    <console type='pty' tty='/dev/pts/3'>
      <source path='/dev/pts/3'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
      <address type='pci' domain='0x0000' bus='0x09' slot='0x01' function='0x0'/>
    </console>

I think this maybe the cause of the issue.

Alright, I've posted some patches here:

https://listman.redhat.com/archives/libvir-list/2023-April/239329.html

After discovering many broken places, I've decided to just forbid live detach of <console/>. I've merged patches upstream as:

8de96e270a qemu_hotplug: Deny live detach of <console/>
b5a591f73b qemuDomainRemoveChrDevice: Deal with qemuDomainChrRemove() failure
fc8320faef qemuAssignDeviceChrAlias: Fix a crasher during <console/> hotplug
e99072731c qemuDomainChrRemove: Don't leak vmdef->consoles[0]
9129643d26 qemuDomainChrInsertPreAlloced: Fix adding implicit console

v9.2.0-253-g8de96e270a

(In reply to Michal Privoznik from comment #11)
> After discovering many broken places, I've decided to just forbid live
> detach of <console/>. I've merged patches upstream as:
> 
> 8de96e270a qemu_hotplug: Deny live detach of <console/>
> b5a591f73b qemuDomainRemoveChrDevice: Deal with qemuDomainChrRemove() failure
> fc8320faef qemuAssignDeviceChrAlias: Fix a crasher during <console/> hotplug
> e99072731c qemuDomainChrRemove: Don't leak vmdef->consoles[0]
> 9129643d26 qemuDomainChrInsertPreAlloced: Fix adding implicit console
> 
> v9.2.0-253-g8de96e270a


I have 2 questions
1. do you mean that forbid live detach of <console> which target type is "serial"? and the detaching of virtio device would not be influenced.  
2. when try to detach the console device with "isa-serial" type, the error message is still " chr type 'console' device not present in domain configuration
". can we make it same with the error message of "pci-serial" device? 



tested on libvirt-9.4.0-1.fc39.x86_64


1. detach a console on isa bus from the guest
[root@zhetang-fedora ~]# virsh dumpxml rhel9-bugverify  --xpath "//console|//serial"
<serial type="pty">
  <source path="/dev/pts/1"/>
  <target type="isa-serial" port="0">
    <model name="isa-serial"/>
  </target>
  <alias name="serial0"/>
</serial>
<console type="pty" tty="/dev/pts/1">
  <source path="/dev/pts/1"/>
  <target type="serial" port="0"/>
  <alias name="serial0"/>
</console>

[root@zhetang-fedora ~]# virsh dumpxml rhel9-bugverify --xpath //console > console.xml
[root@zhetang-fedora ~]# virsh detach-device rhel9-bugverify console.xml 
error: Failed to detach device from console.xml
error: device not found: chr type 'console' device not present in domain configuration



2. detach a console on pci bus from guest
[root@zhetang-fedora ~]# virsh dumpxml rhel9-bugverify  --xpath "//console|//serial"
<serial type="pty">
  <source path="/dev/pts/1"/>
  <target type="pci-serial" port="0">
    <model name="pci-serial"/>
  </target>
  <alias name="serial0"/>
  <address type="pci" domain="0x0000" bus="0x09" slot="0x01" function="0x0"/>
</serial>
<console type="pty" tty="/dev/pts/1">
  <source path="/dev/pts/1"/>
  <target type="serial" port="0"/>
  <alias name="serial0"/>
  <address type="pci" domain="0x0000" bus="0x09" slot="0x01" function="0x0"/>
</console>


[root@zhetang-fedora ~]# virsh dumpxml rhel9-bugverify --xpath //console > console-pci.xml

[root@zhetang-fedora ~]# virsh detach-device rhel9-bugverify console-pci.xml
error: Failed to detach device from console-pci.xml
error: Operation not supported: detaching of <console/> is unsupported. Try corresponding <serial/> instead


3. detach a virtio console
[root@zhetang-fedora ~]# virsh dumpxml rhel9-bugverify --xpath "//console|//serial"
<console type="pty" tty="/dev/pts/1">
  <source path="/dev/pts/1"/>
  <target type="virtio" port="0"/>
  <alias name="console0"/>
</console>


[root@zhetang-fedora ~]# virsh detach-device rhel9-bugverify console-virtio.xml 
Device detached successfully

[root@zhetang-fedora ~]# virsh dumpxml rhel9-bugverify --xpath "//console|//serial"

[root@zhetang-fedora ~]# 



-------------------------------------------------------------------------------------------------

(In reply to zhentang from comment #12)
> (In reply to Michal Privoznik from comment #11)
> > After discovering many broken places, I've decided to just forbid live
> > detach of <console/>. I've merged patches upstream as:
> > 
> > 8de96e270a qemu_hotplug: Deny live detach of <console/>
> > b5a591f73b qemuDomainRemoveChrDevice: Deal with qemuDomainChrRemove() failure
> > fc8320faef qemuAssignDeviceChrAlias: Fix a crasher during <console/> hotplug
> > e99072731c qemuDomainChrRemove: Don't leak vmdef->consoles[0]
> > 9129643d26 qemuDomainChrInsertPreAlloced: Fix adding implicit console
> > 
> > v9.2.0-253-g8de96e270a
> 
> 
> I have 2 questions
> 1. do you mean that forbid live detach of <console> which target type is
> "serial"? and the detaching of virtio device would not be influenced.  

Correct. That should still work.

> 2. when try to detach the console device with "isa-serial" type, the error
> message is still " chr type 'console' device not present in domain
> configuration
> ". can we make it same with the error message of "pci-serial" device? 

Ah, let me see if I can fix it.

(In reply to Michal Privoznik from comment #13)

> Ah, let me see if I can fix it.

BTW: this is so trivial that it should not hold this bug from being verified. If anything, it'll be a trivial patch that we'll pick up by rebase.

Verified on

# rpm -q libvirt
libvirt-9.3.0-2.el9.x86_64
# rpm -q qemu-kvm
qemu-kvm-8.0.0-2.el9.x86_64

Works as expected, error messages could be modified if possible

1. hot unplug isa serial device. remove failed

[root@zhetang-rhel93 hotplug]# virsh start rhel9
Domain 'rhel9' started

[root@zhetang-rhel93 hotplug]# virsh dumpxml rhel9 --xpath //console
<console type="pty" tty="/dev/pts/1">
  <source path="/dev/pts/1"/>
  <target type="serial" port="0"/>
  <alias name="serial0"/>
</console>

[root@zhetang-rhel93 hotplug]# virsh dumpxml rhel9 --xpath //console > serial_console.xml
[root@zhetang-rhel93 hotplug]# virsh detach-device rhel9 serial_console.xml 
error: Failed to detach device from serial_console.xml
error: device not found: chr type 'console' device not present in domain configuration


2. hot unplug pci serial device. remove failed with clear error message. the device can be removed by using <serial> 

[root@zhetang-rhel93 hotplug]# virsh dumpxml rhel9 --xpath //console
<console type="pty" tty="/dev/pts/1">
  <source path="/dev/pts/1"/>
  <target type="serial" port="0"/>
  <alias name="serial0"/>
  <address type="pci" domain="0x0000" bus="0x09" slot="0x01" function="0x0"/>
</console>

[root@zhetang-rhel93 hotplug]# virsh dumpxml rhel9 --xpath //serial
<serial type="pty">
  <source path="/dev/pts/1"/>
  <target type="pci-serial" port="0">
    <model name="pci-serial"/>
  </target>
  <alias name="serial0"/>
  <address type="pci" domain="0x0000" bus="0x09" slot="0x01" function="0x0"/>
</serial>

[root@zhetang-rhel93 hotplug]# virsh dumpxml rhel9 --xpath //console > pci-console.xml
[root@zhetang-rhel93 hotplug]# virsh detach-device rhel9 pci-console.xml 
error: Failed to detach device from pci-console.xml
error: Operation not supported: detaching of <console/> is unsupported. Try corresponding <serial/> instead

[root@zhetang-rhel93 hotplug]# virsh  dumpxml rhel9 --xpath //serial > pci-serial-device.xml
[root@zhetang-rhel93 hotplug]# virsh detach-device rhel9 pci-serial-device.xml 
Device detached successfully


3. remove virtio console device. it could be removed as expected
[root@zhetang-rhel93 hotplug]# virsh dumpxml rhel9 --xpath //console
<console type="pty" tty="/dev/pts/1">
  <source path="/dev/pts/1"/>
  <target type="virtio" port="0"/>
  <alias name="console0"/>
</console>

[root@zhetang-rhel93 hotplug]# virsh dumpxml rhel9 --xpath //console > virtio_console.xml
[root@zhetang-rhel93 hotplug]# virsh detach-device rhel9 virtio_console.xml 
Device detached successfully

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: libvirt security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2023:6409