Bug 1560976 - Hosted Engine VM (deployed in the past) fails to reboot with 'libvirtError: internal error: failed to format device alias for PTY retrieval' due to an error in console device in libvirt XML generated by the engine
Summary: Hosted Engine VM (deployed in the past) fails to reboot with 'libvirtError: i...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: libvirt
Version: 7.5
Hardware: Unspecified
OS: Unspecified
urgent
high
Target Milestone: rc
: ---
Assignee: Michal Privoznik
QA Contact: yalzhang@redhat.com
URL:
Whiteboard:
Depends On: 1560666
Blocks: 1504606 1566525
TreeView+ depends on / blocked
 
Reported: 2018-03-27 11:58 UTC by Michal Skrivanek
Modified: 2018-10-30 09:55 UTC (History)
15 users (show)

Fixed In Version: libvirt-4.3.0-1.el7
Doc Type: Bug Fix
Doc Text:
Previously, user aliases for PTY devices that were longer than 32 characters were not supported. Consequently, if a domain included a PTY device with a user alias longer than 32 characters, the domain would not start. With this update, a static buffer was replaced with a dynamic buffer. As a result, the domain starts even if the length of the user alias for a PTY device is longer than 32 characters.
Clone Of: 1560666
: 1566525 (view as bug list)
Environment:
Last Closed: 2018-10-30 09:53:26 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2018:3113 None None None 2018-10-30 09:55:50 UTC

Description Michal Skrivanek 2018-03-27 11:58:52 UTC
parent bug is implementing temporary workaround (to use old flow skipping all aliases), we'd like to re-enable that as soon as possible

+++ This bug was initially created as a clone of Bug #1560666 +++

Description of problem:
Since 4.2.2, ovirt-ha-agent extracts the XML from libvirt generated by the engine and pass it to VDSM.
In system deployed in the past (with vintage otopi flow), the hosted-engine VM was containing a console device that gets wrongly rendered in that XML.

        <console type="pty">
            <target port="0" type="virtio"/>
            <alias name="ua-c60aba6e-b6d8-448b-ab6e-8c7b5c29f351"/>
        </console>

Systems deployed with the ansible flow are not affected since the engine VM got create by the engine via REST APIs.


Version-Release number of selected component (if applicable):
4.2.2-rc5 (beta3)

How reproducible:
?

Steps to Reproduce:
1. deploy hosted-engine with the vintage flow
2. wait for the engine to create OVF_STORE disks
3. try to restart the engine VM

Actual results:
the XML generated from the engine contains
        <console type="pty">
            <target port="0" type="virtio"/>
            <alias name="ua-c60aba6e-b6d8-448b-ab6e-8c7b5c29f351"/>
        </console>

the VM fails to start with:
2018-03-26 11:54:31,272-0400 ERROR (vm/eccb9cb6) [virt.vm] (vmId='eccb9cb6-affd-4806-8200-708370581227') The vm start process failed (vm:940)
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 869, in _startUnderlyingVm
    self._run()
  File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2832, in _run
    dom.createWithFlags(flags)
  File "/usr/lib/python2.7/site-packages/vdsm/common/libvirtconnection.py", line 130, in wrapper
    ret = f(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/vdsm/common/function.py", line 92, in wrapper
    return func(inst, *args, **kwargs)
  File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1099, in createWithFlags
    if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
libvirtError: internal error: failed to format device alias for PTY retrieval

Expected results:
The engine VM is able to restart from systems deployed in the past also booting from the XML created by the engine.

Additional info:
In only affects systems deployed with the vintage flow

Comment 1 yalzhang@redhat.com 2018-03-28 03:21:02 UTC
I can reproduce it on libvirt-3.9.0-14.el7_5.2.x86_64, it seems that the alias name only accept 24 characters

For 24 characters, guest start successfully
# virsh dumpxml rhel
...
<console type='pty'>
      <source path='/dev/pts/4'/>
      <target type='virtio' port='1'/>
      <alias name='ua-123456789012345678901234'/>
    </console>
...
# virsh start rhel
Domain rhel started

For 25 characters, vm can not start
# virsh dumpxml rhel
...
<console type='pty'>
      <target type='virtio' port='1'/>
      <alias name='ua-1234567890123456789012345'/>
    </console>
...
# virsh start rhel
error: Failed to start domain rhel
error: internal error: failed to format device alias for PTY retrieval

Comment 2 yalzhang@redhat.com 2018-03-28 06:49:44 UTC
1. not only for virtio pty console and serial pty console and serial devices, all virtio channel do have such issue, including virtio channel with type=pty/vc/dev/file/pipe/stdio/udp/tcp/unix/spicevmc

# virsh dumpxml rhel | grep ua -A2 -B2
    <channel type='unix'>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <alias name='ua-c60aba6e-b6d8-448b-ab6e-8c7b5c29f353'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>

# virsh start rhel
error: Failed to start domain rhel
error: internal error: failed to format device alias for PTY retrieval

And some libvirtd.log for reference:

2018-03-28 03:34:40.927+0000: 8528: error : qemuProcessLookupPTYs:1946 : internal error: failed to format device alias for PTY retrieval
...
2018-03-28 04:54:59.774+0000: 8527: error : qemuProcessRefreshChannelVirtioState:2037 : internal error: failed to format device alias for PTY retrieval

2. channel with target type is guestfwd will ignore any alias setting.
# virsh edit rhel ===> to add alias like below
 <channel type='pty'>
      <target type='guestfwd' address='10.0.2.1' port='4600'/>
     <alias name='ua-justfortest'/>
    </channel>

# virsh dumpxml rhel
...
<channel type='pty'>
      <target type='guestfwd' address='10.0.2.1' port='4600'/>
    </channel>

==> no alias, the alias setting is ignored for channel with guestfwd target type


3. If we will add the alias name length check? I have tried a little, have not found the max length

1) set  <alias name='ua-'/> for interface device, vm can start with qemu option:

 -netdev tap,fd=27,id=hostua-,vhost=on,vhostfd=29 -device virtio-net-pci,netdev=hostua-,id=ua-,mac=52:54:00:03:20:4c,bus=pci.0,addr=0x3


2) tried with alias name as 'ua-{9027 characters}' with interface device, vm can start successfully

Comment 3 Michal Privoznik 2018-03-29 06:52:46 UTC
Patch proposed upstream:

https://www.redhat.com/archives/libvir-list/2018-March/msg01806.html

Comment 4 Michal Privoznik 2018-04-01 10:41:13 UTC
I've just pushed the patch upstream:

commit c4c32cb300beba41ecbca3ee4884c65630bde861
Author:     Michal Privoznik <mprivozn@redhat.com>
AuthorDate: Thu Mar 29 08:50:01 2018 +0200
Commit:     Michal Privoznik <mprivozn@redhat.com>
CommitDate: Sun Apr 1 12:38:38 2018 +0200

    qemu: Use dynamic buffer for storing PTY aliases
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1560976
    
    For historical reasons we've used 32 bytes long static buffer for
    storing PTY aliases. This breaks users scenario where they try to
    start a machine with user alias consisting of "ua-$uuid".
    
    Signed-off-by: Michal Privoznik <mprivozn@redhat.com>

v4.2.0-2-gc4c32cb300

Comment 7 yalzhang@redhat.com 2018-05-30 03:20:10 UTC
test on libvirt-4.3.0-1.el7.x86_64, the result is as expected.

1. Set alias name for serial device in vm xml:
# virsh dumpxml rhel | grep /console -B9
    <serial type='pty'>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
      <alias name='ua-3ea66e7d-40f6-4396-a3c0-3160c9bfdabe'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
      <alias name='ua-3ea66e7d-40f6-4396-a3c0-3160c9bfdabe'/>
    </console>
......
 <channel type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/domain-8-rhel/org.qemu.guest_agent.0'/>
      <target type='virtio' name='org.qemu.guest_agent.0' state='disconnected'/>
      <alias name='ua-62e41900-52d9-4fe7-b788-c04dbb8e61e9'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>

2. Start the vm
# virsh start rhel
Domain rhel started

3. Hotplug a virtio console, pci serial, virtio channel, then hot-unplug, all succeed
# cat console.xml
<console type='dev'>        
 <source path='/dev/ttyS0'/>      
 <target type='virtio' port='1'/> 
<alias name='ua-99d9ca12-dbd1-4c3f-b9c4-cfcccdad86aa'/>
</console>
# virsh attach-device rhel console.xml
Device attached successfully

# cat serial.xml
<serial type='pty'>
 <source path='/dev/pts/4'/>
 <target type='pci-serial' port='0'/> 
<alias name='ua-89a37c8e-363a-4936-b84d-863191eed07e'/>
</serial>
# virsh attach-device rhel serial.xml
Device attached successfully

# cat channel.xml
<channel type='pty'>       
<source path='/dev/pts/5'/>      
 <target type='virtio' name='arbitrary.virtio.serial.port.name'/>   
    <alias name='ua-9b8ebb59-093d-4f9c-b467-99a433e278a7'/>       
 </channel>
# virsh attach-device rhel channel.xml
Device attached successfully

# virsh dumpxml rhel 
==> check all devices exists with the alias name set

4. test duplicate alias
# virsh attach-device rhel ch.xml
error: Failed to attach device from ch.xml
error: XML error: non unique alias detected: ua-3ea66e7d-40f6-4396-a3c0-3160c9bfdabe

Comment 9 errata-xmlrpc 2018-10-30 09:53:26 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:3113


Note You need to log in before you can comment on or make changes to this bug.