Bug 1180085

Summary: ovirt-hosted-engine-setup failed to execute stage 'Misc configuration': Failed to setup networks
Product: Red Hat Enterprise Virtualization Manager Reporter: Petr Matyáš <pmatyas>
Component: vdsmAssignee: Nobody <nobody>
Status: CLOSED WORKSFORME QA Contact: Meni Yakove <myakove>
Severity: high Docs Contact:
Priority: unspecified    
Version: 3.5.0CC: bazulay, danken, ecohen, gklein, iheim, lpeer, lsurette, oourfali, pmatyas, sbonazzo, yeylon
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: network
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-01-29 11:12:50 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Network RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1105479    
Attachments:
Description Flags
VDSM log
none
supervdsm log
none
libvirtd configuration file
none
libvirtd log
none
new vdsm log
none
VDSM log
none
supervdsm log none

Description Petr Matyáš 2015-01-08 10:18:14 UTC
Created attachment 977714 [details]
VDSM log

Description of problem:
When setting up ovirt-hosted-engine I get [ ERROR ] Failed to execute stage 'Misc configuration': Failed to setup networks {'rhevm': {'nic': 'eth0', 'netmask': '255.255.252.0', 'bootproto': 'none', 'ipaddr': '10.34.60.233'}}. Error code: "16" message: "Unexpected exception"

Version-Release number of selected component (if applicable):
vt13.6

How reproducible:
always on my setup

Steps to Reproduce:
1. set BOOTPROTO=none and comment out/remove GATEWAY in network config
2. run ovirt-hosted-engine-setup
3. use edited network

Actual results:
error

Expected results:
successful setup

Additional info:

Comment 2 Sandro Bonazzola 2015-01-20 09:46:25 UTC
 Thread-15::DEBUG::2015-01-08 10:26:42,134::BindingXMLRPC::1142::vds::(wrapper)  client [127.0.0.1]::call setupNetworks with ({'rhevm': {'nic': 'eth0', 'netmask': '255.255.252.0', 'bootproto': 'none', 'ipaddr': '10.34.60.233'}}, {}, {'connectivityCheck': False}) {}
 Thread-15::ERROR::2015-01-08 10:26:42,210::BindingXMLRPC::1161::vds::(wrapper) unexpected error
 Traceback (most recent call last):
  File "/usr/share/vdsm/rpc/BindingXMLRPC.py", line 1145, in wrapper
    res = f(*args, **kwargs)
  File "/usr/share/vdsm/rpc/BindingXMLRPC.py", line 563, in setupNetworks
    return api.setupNetworks(networks, bondings, options)
  File "/usr/share/vdsm/API.py", line 1385, in setupNetworks
    supervdsm.getProxy().setupNetworks(networks, bondings, options)
  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
    return callMethod()
  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
    **kwargs)
  File "<string>", line 2, in setupNetworks
  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 726, in _callmethod
    kind, result = conn.recv()
 EOFError

Looks like an issue within VDSM. moving to that component.

Comment 3 Dan Kenigsberg 2015-01-21 12:45:47 UTC
It seems that supervdsm crashed while setting up the network. Please attach supervdsm.log

Comment 4 Petr Matyáš 2015-01-21 13:22:54 UTC
Created attachment 982327 [details]
supervdsm log

Comment 5 Dan Kenigsberg 2015-01-22 14:15:28 UTC
The plot thickens. supervdsm took itself down since its connection to libvirt was broken.

MainProcess|Thread-15::CRITICAL::2015-01-08 10:26:42,140::libvirtconnection::137::root::(wrapper) taking calling process down.
MainThread::DEBUG::2015-01-08 10:26:42,140::supervdsmServer::453::SuperVdsm.Server::(main) Terminated normally
MainProcess|Thread-15::DEBUG::2015-01-08 10:26:42,142::libvirtconnection::143::root::(wrapper) Unknown libvirterror: ecode: 1 edom: 7 level: 2 message: internal error client socket is closed

Could you attach libvirtd.conf, libvirtd.log and supply libvirt's version?

Comment 6 Dan Kenigsberg 2015-01-22 14:17:01 UTC
Does /var/log/message has any trace of a libvirtd crash?

Comment 7 Petr Matyáš 2015-01-22 14:29:21 UTC
Created attachment 982841 [details]
libvirtd configuration file

Comment 8 Petr Matyáš 2015-01-22 14:29:46 UTC
Created attachment 982843 [details]
libvirtd log

Comment 9 Petr Matyáš 2015-01-22 14:33:32 UTC
libvirt-0.10.2-46.el6_6.2.x86_64

/var/log/messages doesn't show any error or crash from that day

Comment 10 Dan Kenigsberg 2015-01-22 16:15:47 UTC
Is this libvirtd.log in its entirety?

2015-01-21 12:16:21.097+0000: 1566: error : virNetSocketNewListenTCP:239 : Unable to resolve address '0.0.0.0' service '16514': Address family for hostname not supported

if so, could you set

 log_outputs="1:file:/var/log/libvirt/libvirtd.log"
 auth_unix_rw="none"

in libvirtd.conf, pkill libvirtd, and try to reproduce?
Before installation, does `virsh list` succeed?

Comment 11 Petr Matyáš 2015-01-23 11:30:09 UTC
[root@pm-upgrade2 ~]# virsh list
error: Failed to reconnect to the hypervisor
error: no valid connection
error: Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory

When I changed the two lines I couldn't reproduce this bug, although I found a different one:
[ ERROR ] Failed to execute stage 'Closing up': Cannot set temporary password for console connection. The VM may not have been created: please check VDSM logs

Comment 12 Petr Matyáš 2015-01-23 11:30:36 UTC
Created attachment 983306 [details]
new vdsm log

Comment 13 Dan Kenigsberg 2015-01-24 22:36:28 UTC
error: Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory

means that libvirtd failed to start. we should debug this issue first - does the more verbose libvirtd.log say anything?

Comment 14 Petr Matyáš 2015-01-28 13:49:48 UTC
I tried to reproduce this issue but I couldn't, the virsh list now succeeds although it prints no domains. However I reproduced the issue I found before (comment #11)

Comment 15 Dan Kenigsberg 2015-01-28 14:13:24 UTC
Could you supply the logs of the new reproduction?

Comment 16 Petr Matyáš 2015-01-28 14:26:57 UTC
Created attachment 985194 [details]
VDSM log

Comment 17 Petr Matyáš 2015-01-28 14:27:42 UTC
Created attachment 985195 [details]
supervdsm log

Comment 18 Dan Kenigsberg 2015-01-28 14:56:56 UTC
It seems that qemu process crashed right after startup. Can you attach qemu and libvirt logs? Could it be that your host does not have enough ram+swap space for this VM?

Thread-77::DEBUG::2015-01-28 13:22:08,363::libvirtconnection::143::root::(wrapper) Unknown libvirterror: ecode: 1 edom: 10 level: 2 message: internal error Process exited while reading console log output: char device redirected to /dev/pts/2
Cannot set up guest memory 'pc.ram': Cannot allocate memory

Comment 19 Petr Matyáš 2015-01-29 11:12:50 UTC
Sorry, it really didn't have enough memory. Must have been fixed with something else because it works fine now.