Bug 253371

Summary: libvirt fails with hotplug error (32bits on 64bits paravirt guest both fc7)
Product: [Fedora] Fedora Reporter: Adam Greenberg <moose>
Component: python-virtinstAssignee: Cole Robinson <crobinso>
Status: CLOSED WORKSFORME QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: high Docs Contact:
Priority: high    
Version: 7CC: triage
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2008-05-20 20:13:08 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Creation log files.
none
console iimage
none
ubuntu feisty screen shot. none

Description Adam Greenberg 2007-08-18 12:56:35 UTC
Description of problem:
libvirt fails with a hotplug error after creating a domain.


Version-Release number of selected component (if applicable):
libvirt-0.3.1-2.fc7.x86-64

How reproducible:
Reliably reproducible (all attempts)


Steps to Reproduce:
1.  Start virtual machine creator
2.  Select paravirtualized guest using the fedora fc7 (32 bit) image from red hat.
3.  Click finish and watch the process fail after creating the domain.

Note:  If I select the static network option, the process fails with a blank
console appearing.  If I select the non-static option, I get a console that I
can interact with but the process then fails in the network set up with the
result noted below.


  
Actual results:

Unable to complete install '<class 'libvirt.libvirtError'>
virDomainCreateLinux() failed POST operation failed: (xend.err 'Device 0 (vif)
could not be connected. Hotplug scripts not working.')
Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/create.py", line 681, in do_install
    dom = guest.start_install(False, meter = meter)
  File "/usr/lib/python2.5/site-packages/virtinst/Guest.py", line 710, in
start_install
    return self._do_install(consolecb, meter)
  File "/usr/lib/python2.5/site-packages/virtinst/Guest.py", line 727, in
_do_install
    self.domain = self.conn.createLinux(install_xml, 0)
  File "/usr/lib64/python2.5/site-packages/libvirt.py", line 525, in createLinux
    if ret is None:raise libvirtError('virDomainCreateLinux() failed', conn=self)
libvirtError: virDomainCreateLinux() failed POST operation failed: (xend.err
'Device 0 (vif) could not be connected. Hotplug scripts not working.')
'


Expected results:
A continuing installation process for the new guest.

Additional info:
Domain builder log output:
Unable to complete install '<class 'libvirt.libvirtError'>
virDomainCreateLinux() failed POST operation failed: (xend.err 'Device 0 (vif)
could not be connected. Hotplug scripts not working.')
Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/create.py", line 681, in do_install
    dom = guest.start_install(False, meter = meter)
  File "/usr/lib/python2.5/site-packages/virtinst/Guest.py", line 710, in
start_install
    return self._do_install(consolecb, meter)
  File "/usr/lib/python2.5/site-packages/virtinst/Guest.py", line 727, in
_do_install
    self.domain = self.conn.createLinux(install_xml, 0)
  File "/usr/lib64/python2.5/site-packages/libvirt.py", line 525, in createLinux
    if ret is None:raise libvirtError('virDomainCreateLinux() failed', conn=self)
libvirtError: virDomainCreateLinux() failed POST operation failed: (xend.err
'Device 0 (vif) could not be connected. Hotplug scripts not working.')
'

Comment 1 Daniel Veillard 2007-08-20 09:12:09 UTC
Can you confirm it's a 64bits setup where you try to install 32bits F-7 ? If yes,
Paravirt 32bits on 64bits is somehow a work in progress, Use 64on64 if you want
reliability now on that hardware. This is not a libvirt errror, but a Xen problem,
libvirt just merely try to report the errors occuring in the hypervisor.
Provide the XML configuration for the domain if you have it.

Daniel

Comment 2 Adam Greenberg 2007-09-07 02:08:33 UTC
I was able to create and run a paravirtualized FC7-64 guest.  There is
documentation (I found it on the web) that indicates that 32 bit guests on 64
bit hosts are not supported in the Xen version upon which the Fedora 7 kernel is
built.  Perhaps this will be fixed in FC8.  Now I find that I cannot create even
64 bit guests using full virtualization.  I get a vncpasswd error every time. 
I've described this in bug 251441
https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=251441, but have not seen
any investigation of this yet (no response to this but other than to copy it to
the xen-maint list).

Comment 3 Daniel Berrangé 2007-09-07 02:53:11 UTC
Please provide the /var/log/xen/xen.log and 
/root/.virt-manager/virt-manager.log from just after the failed VM creation attempt.

Comment 4 Daniel Berrangé 2007-09-07 02:56:40 UTC
FYI, the error message about vncpasswd is a bogus message - virt-install doesn't
configure any password, so it is expected that qemu can't read one ! The
xend.log file should help diagnose the root problem.


Comment 5 Adam Greenberg 2007-09-07 03:06:19 UTC
Ok.  By moving the iso image to /var/lib/xen/images, I get past an selinux
protection error.  Now the console comes up but the guest process dies.  Here is
the virt-manager log:

Thu, 06 Sep 2007 22:52:21 DEBUG    OS Type: linux
Thu, 06 Sep 2007 22:52:21 DEBUG    OS Variant: sles10
Thu, 06 Sep 2007 22:52:21 DEBUG    Creating a VM suse
  Type: xen
  UUID: 833e6421-7eb2-d8a8-480f-cdc501214b0d
  Source: /var/lib/xen/images/openSUSE-10.2-GM-x86_64-mini.iso
  OS: Suse Linux Enterprise Server
  Kickstart: 
  Memory: 500
  Max Memory: 1500
  # VCPUs: 1
  Filesize: 3.90625
  Disk image: /var/lib/xen/images/suse
  Non-sparse file: False
Thu, 06 Sep 2007 22:52:21 DEBUG    Sparse file or partition selected
Thu, 06 Sep 2007 22:52:21 DEBUG    Starting background install process
Thu, 06 Sep 2007 22:52:21 DEBUG    Creating guest from '<domain type='xen'>
  <name>suse</name>
  <currentMemory>512000</currentMemory>
  <memory>1536000</memory>
  <uuid>833e6421-7eb2-d8a8-480f-cdc501214b0d</uuid>
  <os>
    <type>hvm</type>
    <loader>/usr/lib/xen/boot/hvmloader</loader>
    <boot dev='cdrom'/>
  </os>
  <features>
    <apic/><acpi/><pae/>
  </features>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>destroy</on_reboot>
  <on_crash>destroy</on_crash>
  <vcpu>1</vcpu>
  <devices>
    <emulator>/usr/lib64/xen/bin/qemu-dm</emulator>
    <console device='pty'/>
    <disk type='file' device='disk'>
      <source file='/var/lib/xen/images/suse'/>
      <target dev='hda'/>
    </disk>
    <disk type='file' device='cdrom'>
      <source file='/var/lib/xen/images/openSUSE-10.2-GM-x86_64-mini.iso'/>
      <target dev='hdc'/>
      <readonly/>
    </disk>

    <interface type='bridge'>
      <source bridge='eth0'/>
      <mac address='00:16:3e:3d:9b:31'/>
    </interface>

    <graphics type='vnc' port='-1' keymap='en-us' />
  </devices>
</domain>
'
Thu, 06 Sep 2007 22:52:22 DEBUG    VM suse started
Thu, 06 Sep 2007 22:52:23 DEBUG    Created guest, looking to see if it is running
Thu, 06 Sep 2007 22:52:24 DEBUG    Saving XML boot config '<domain type='xen'>
  <name>suse</name>
  <currentMemory>512000</currentMemory>
  <memory>1536000</memory>
  <uuid>833e6421-7eb2-d8a8-480f-cdc501214b0d</uuid>
  <os>
    <type>hvm</type>
    <loader>/usr/lib/xen/boot/hvmloader</loader>
    <boot dev='hd'/>
  </os>
  <features>
    <apic/><acpi/><pae/>
  </features>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <vcpu>1</vcpu>
  <devices>
    <emulator>/usr/lib64/xen/bin/qemu-dm</emulator>
    <console device='pty'/>
    <disk type='file' device='disk'>
      <source file='/var/lib/xen/images/suse'/>
      <target dev='hda'/>
    </disk>
    <disk type='file' device='cdrom'>
      <source file=''/>
      <target dev='hdc'/>
      <readonly/>
    </disk>

    <interface type='bridge'>
      <source bridge='eth0'/>
      <mac address='00:16:3e:3d:9b:31'/>
    </interface>

    <graphics type='vnc' port='-1' keymap='en-us' />
  </devices>
</domain>
'
Thu, 06 Sep 2007 22:52:24 DEBUG    Install completed
Thu, 06 Sep 2007 22:52:30 DEBUG    Graphics console configured at
vnc://127.0.0.1:5901
Thu, 06 Sep 2007 22:52:30 DEBUG    Graphics console configured at
vnc://127.0.0.1:5901

However, the qemu log still reports the vncpasswd error.  I do not get a xen.log.

Comment 6 Daniel Berrangé 2007-09-07 03:20:30 UTC
Ah ha. The second piece of XML in there is bogus - the <source file=''>  tag 
should not be present at all. Can you test the -2.fc7.1dantest custom RPM build
with an experimental patch from this URL:

http://fedorapeople.org/~berrange/python-virtinst/

Hopefully it shoud fix the XML for the post-install case.

Second, I typod for the logfile name - I meant /var/log/xen/xend.log  not xen.log

Comment 7 Adam Greenberg 2007-09-07 03:32:57 UTC
I tried to install your patch rpm.  The installation fails with a missing
dependency though - python(abi) = 2.4.  Is there a preferred way to install this?

I don't get a /var/log/xen/xend.log either.  I only get qemu and xen-hotplug.

Comment 8 Daniel Berrangé 2007-09-07 03:48:36 UTC
Sorry, I built them on the wrong machine - FC6 instead of F7. Try the newer
-2.fc7.2dantest RPMs instead from the same site.


Comment 9 Adam Greenberg 2007-09-07 04:01:58 UTC
In installed your latest patch.  The guest creation appears to fail the same
way.  Here is the .virtinst log:

Thu, 06 Sep 2007 23:58:52 DEBUG    OS Type: linux
Thu, 06 Sep 2007 23:58:52 DEBUG    OS Variant: sles10
Thu, 06 Sep 2007 23:58:52 DEBUG    Creating a VM suse
  Type: xen
  UUID: c68904f5-fe02-6e65-2008-efeed21f8cca
  Source: /var/lib/xen/images/openSUSE-10.2-GM-x86_64-mini.iso
  OS: Suse Linux Enterprise Server
  Kickstart: 
  Memory: 500
  Max Memory: 1500
  # VCPUs: 1
  Filesize: 3.90625
  Disk image: /var/lib/xen/images/suse
  Non-sparse file: True
Thu, 06 Sep 2007 23:58:52 DEBUG    Non-sparse file selected
Thu, 06 Sep 2007 23:58:52 DEBUG    Starting background install process
Thu, 06 Sep 2007 23:58:52 DEBUG    Creating guest from '<domain type='xen'>
  <name>suse</name>
  <currentMemory>512000</currentMemory>
  <memory>1536000</memory>
  <uuid>c68904f5-fe02-6e65-2008-efeed21f8cca</uuid>
  <os>
    <type>hvm</type>
    <loader>/usr/lib/xen/boot/hvmloader</loader>
    <boot dev='cdrom'/>
  </os>
  <features>
    <apic/><acpi/><pae/>
  </features>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>destroy</on_reboot>
  <on_crash>destroy</on_crash>
  <vcpu>1</vcpu>
  <devices>
    <emulator>/usr/lib64/xen/bin/qemu-dm</emulator>
    <console device='pty'/>
    <disk type='file' device='disk'>
      <source file='/var/lib/xen/images/suse'/>
      <target dev='hda'/>
    </disk>
    <disk type='file' device='cdrom'>
      <source file='/var/lib/xen/images/openSUSE-10.2-GM-x86_64-mini.iso'/>
      <target dev='hdc'/>
      <readonly/>
    </disk>

    <interface type='bridge'>
      <source bridge='eth0'/>
      <mac address='00:16:3e:74:e9:51'/>
    </interface>

    <graphics type='vnc' port='-1' keymap='en-us' />
  </devices>
</domain>
'
Thu, 06 Sep 2007 23:58:53 DEBUG    VM suse started
Thu, 06 Sep 2007 23:58:54 DEBUG    Created guest, looking to see if it is running
Thu, 06 Sep 2007 23:58:54 DEBUG    Saving XML boot config '<domain type='xen'>
  <name>suse</name>
  <currentMemory>512000</currentMemory>
  <memory>1536000</memory>
  <uuid>c68904f5-fe02-6e65-2008-efeed21f8cca</uuid>
  <os>
    <type>hvm</type>
    <loader>/usr/lib/xen/boot/hvmloader</loader>
    <boot dev='hd'/>
  </os>
  <features>
    <apic/><acpi/><pae/>
  </features>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <vcpu>1</vcpu>
  <devices>
    <emulator>/usr/lib64/xen/bin/qemu-dm</emulator>
    <console device='pty'/>
    <disk type='file' device='disk'>
      <source file='/var/lib/xen/images/suse'/>
      <target dev='hda'/>
    </disk>
    <disk type='file' device='cdrom'>
      <target dev='hdc'/>
      <readonly/>
    </disk>

    <interface type='bridge'>
      <source bridge='eth0'/>
      <mac address='00:16:3e:74:e9:51'/>
    </interface>

    <graphics type='vnc' port='-1' keymap='en-us' />
  </devices>
</domain>
'
Thu, 06 Sep 2007 23:58:54 DEBUG    Install completed
Thu, 06 Sep 2007 23:58:54 DEBUG    Graphics console configured at
vnc://127.0.0.1:5901
Thu, 06 Sep 2007 23:58:54 WARNING  Keyring unavailable: '<class
'gnomekeyring.NoKeyringDaemonError'> '
Thu, 06 Sep 2007 23:58:54 DEBUG    Graphics console configured at
vnc://127.0.0.1:5901


Comment 10 Adam Greenberg 2007-09-07 11:31:34 UTC
Here is another bit of information.  When I use the guest creation sizard, and
select "fully virtualized", the panel keeps the cpu architecture and "Enable
kernel / hardware accelleration" check box grayed out.  Perhaps this is significant?

Comment 11 Daniel Berrangé 2007-09-07 17:09:46 UTC
> I don't get a /var/log/xen/xend.log either.  I only get qemu and xen-hotplug.

This is a major problem. You absolutely should get a xend.log.  Do you have
SELinux running in 'enforcing' mode ? If so, edit /etc/sysconfig/selinux, switch
it to permissive, *reboot* to ensure all Xen daemons are correctly restarted and
then try to create the guest again. I need the xend.log to be able to debug this
any further.

Finally, do you have Intel, or AMD CPUs ?  SUSE has a graphical boot screen
which sometimes causes problems with Intel CPUs so you might need to disable
that. Not sure how todo it offhand though.

Comment 12 Adam Greenberg 2007-09-07 19:54:58 UTC
Created attachment 190421 [details]
Creation log files.

I changed SELinux to permissive and rebooted.  Here are the logs of the
creation attempt.

Comment 13 Adam Greenberg 2007-09-07 19:58:56 UTC
Created attachment 190431 [details]
console iimage

My machine is a quad core Intel box.  Here is a snapshot of the guest console
-as far as it goes.

Comment 14 Daniel Berrangé 2007-09-07 20:12:32 UTC
That screenshot shows that the inital bootloader is hanging. This is a known
problem with SUSE and Ubuntu distributuons when running on Intel CPUs, since
Intel lacks 16-bit realmode virtualization.

Googling around the solutions are to either hold down SHIFT while the CD is
booting - pretty hard since it boots very quickly, or get a boot ISO without the
fancy graphical splash screen at boot - for Ubuntu it seems the mini network
boot ISO does the job - not sure if SUSE has an equivalent.

http://dev.osso.nl/herman/blog/2006/12/24/ubuntu-bootmenu-too-shiny-for-xen-with-hvm/


Comment 15 Adam Greenberg 2007-09-07 20:38:37 UTC
Hmmm.  This is the network load iso for suse 10.2.  Sigh.  Thanks for your help.

Comment 16 Adam Greenberg 2007-09-07 22:06:47 UTC
I tried the debian 4 network boot and it fails the same way.  I suspect a deeper
problem.

Comment 17 Adam Greenberg 2007-09-08 01:07:57 UTC
This fails with Fedora net boot too.  I strongly suspect that this problem is
unrelated to the Intel VT 16 bit issue as it appears to happen regardless of
linux variant.

Comment 18 Adam Greenberg 2007-09-08 01:32:08 UTC
Created attachment 190641 [details]
ubuntu feisty screen shot.

I tired the ubuntu edgy mini.iso from the url in the web page you noted above. 
That seems to load.  I then tried the feisty version and that failed in the
boot loader.  Here is a screen shot.

Comment 19 Cole Robinson 2008-03-12 19:47:35 UTC
Hi, are you still having issues with booting a fedora guest using the boot.iso
with the latest {xen, kernel-xen, libvirt} packages?

Comment 20 Bug Zapper 2008-05-14 14:01:29 UTC
This message is a reminder that Fedora 7 is nearing the end of life. Approximately 30 (thirty) days from now Fedora will stop maintaining and issuing updates for Fedora 7. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as WONTFIX if it remains open with a Fedora 'version' of '7'.

Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version prior to Fedora 7's end of life.

Bug Reporter: Thank you for reporting this issue and we are sorry that we may not be able to fix it before Fedora 7 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora please change the 'version' of this bug. If you are unable to change the version, please add a comment here and someone will do it for you.

Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete. If possible, it is recommended that you try the newest available Fedora distribution to see if your bug still exists.

Please read the Release Notes for the newest Fedora distribution to make sure it will meet your needs:
http://docs.fedoraproject.org/release-notes/

The process we are following is described here: http://fedoraproject.org/wiki/BugZappers/HouseKeeping

Comment 21 Cole Robinson 2008-05-20 20:13:08 UTC
This bug has covered a lot of ground, and since nobody is reporting similar
errors with the current versions of virt-install, and there has been no response
for 2 months, I'm closing this WORKSFORME.

Comment 22 Adam Greenberg 2008-05-21 00:00:26 UTC
I no longer run FC7 so I cannot test this.  Many thanks for checking though.