Bug 1036882

Summary: Waiting to start VDSM
Product: [Retired] oVirt Reporter: Maurice James <midnightsteel>
Component: vdsmAssignee: Yaniv Bronhaim <ybronhei>
Status: CLOSED NOTABUG QA Contact: Haim <hateya>
Severity: medium Docs Contact:
Priority: urgent    
Version: 3.3CC: abaron, acathrow, bazulay, iheim, mgoldboi, midnightsteel, yeylon
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard: infra
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-12-23 00:34:57 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Attachments:
Description Flags
host-deploy-logs none

Description Maurice James 2013-12-02 20:18:30 UTC
Description of problem:
Host get stuck starting VDSM because of permission on "/var/run/libvirt/libvirt-sock"

Version-Release number of selected component (if applicable):
VDSM-4.13.0-11
ovirt-engine 3.3

How reproducible:


Steps to Reproduce:
1. Install new instance of Ovirt
2. Run engine-setup
3. Wait for it to complete

Actual results:
Installation completes but VDSM fails to start

Expected results:
installation completes without error

Additional info:

Comment 1 Dan Kenigsberg 2013-12-02 22:05:43 UTC
Could you attach ovirt-host-deploy logs?

Which is your underlying platform (Fedora/EL6)? Do you see any hints for the vdsm startup failure in /var/log/messages?

Comment 2 Maurice James 2013-12-02 23:49:41 UTC
Created attachment 831841 [details]
host-deploy-logs

The host-deploy log is attached
Here is a snip from the messages log:

Dec  2 15:37:28 beetlejuice vdsm TaskManager.Task ERROR Task=`639d0236-2b89-4c95-81c4-d864987aa246`::Unexpected error#012Traceback (most recent call last):#012  File "/usr/share/vdsm/storage/task.py", line 857, in _run#012    return fn(*args, **kargs)#012  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper#012    res = f(*args, **kwargs)#012  File "/usr/share/vdsm/storage/hsm.py", line 615, in getSpmStatus#012    pool = self.getPool(spUUID)#012  File "/usr/share/vdsm/storage/hsm.py", line 318, in getPool#012    raise se.StoragePoolUnknown(spUUID)#012StoragePoolUnknown: Unknown pool id, pool not connected: ('5849b030-626e-47cb-ad90-3ce782d831b3',)
Dec  2 15:37:28 beetlejuice kernel: ata1: EH complete Dec  2 15:37:28 beetlejuice kernel: ata2: EH complete Dec  2 15:37:28 beetlejuice kernel: ata3: soft resetting link Dec  2 15:37:28 beetlejuice kernel: ata4: soft resetting link Dec  2 15:37:28 beetlejuice kernel: ata3.00: configured for UDMA/133 Dec  2 15:37:28 beetlejuice kernel: ata3: EH complete Dec  2 15:37:28 beetlejuice kernel: ata4.00: configured for UDMA/100 Dec  2 15:37:28 beetlejuice kernel: ata4: EH complete Dec  2 15:37:30 beetlejuice kernel: device-mapper: table: 253:3: multipath: error getting device Dec  2 15:37:30 beetlejuice kernel: device-mapper: ioctl: error adding target to table Dec  2 15:39:28 beetlejuice ntpd[2276]: 0.0.0.0 061c 0c clock_step +0.337626 s Dec  2 15:39:28 beetlejuice ntpd[2276]: 0.0.0.0 0615 05 clock_sync Dec  2 15:39:29 beetlejuice ntpd[2276]: 0.0.0.0 c618 08 no_sys_peer Dec  2 16:26:57 beetlejuice kernel: device vnet0 entered promiscuous mode Dec  2 16:26:57 beetlejuice kernel: ovirtmgmt: port 2(vnet0) entering forwarding state Dec  2 16:26:58 beetlejuice kernel: ip_tables: (C) 2000-2006 Netfilter Core Team Dec  2 16:26:58 beetlejuice kernel: ovirtmgmt: port 2(vnet0) entering disabled state Dec  2 16:26:58 beetlejuice kernel: device vnet0 left promiscuous mode Dec  2 16:26:58 beetlejuice kernel: ovirtmgmt: port 2(vnet0) entering disabled state
Dec  2 16:26:58 beetlejuice vdsm vm.Vm ERROR vmId=`16e87c5f-ffbb-4ac8-a18f-41959347cd5e`::The vm start process failed#012Traceback (most recent call last):#012  File "/usr/share/vdsm/vm.py", line 2092, in _startUnderlyingVm#012    self._run()#012  File "/usr/share/vdsm/vm.py", line 2959, in _run#012    self._connection.createXML(domxml, flags),#012  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 76, in wrapper#012    ret = f(*args, **kwargs)#012  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 2665, in createXML#012    if ret is None:raise libvirtError('virDomainCreateXML() failed', conn=self)#012libvirtError: internal error ifname "vnet0" not in key map
Dec  2 16:27:00 beetlejuice vdsm vm.Vm WARNING vmId=`16e87c5f-ffbb-4ac8-a18f-41959347cd5e`::trying to set state to Powering down when already Down

Comment 3 Yaniv Bronhaim 2013-12-10 13:47:28 UTC
please attach also vdsm.log and /var/log/message to the bug. where did you see the permission error? during the start? is it reproducible?

it should be:
srwxrwx---. 1 root kvm 0 Dec  9 23:46 /var/run/libvirt/libvirt-sock

can you copy the output of "ls -la /var/run/libvirt/libvirt-sock" ?

Thanks

Comment 4 Yaniv Bronhaim 2013-12-12 17:34:21 UTC
Hey Maurice James, Can you provide the information to help investigate the issue?

Comment 5 Yaniv Bronhaim 2013-12-17 14:19:18 UTC
ping?

Comment 6 Maurice James 2013-12-23 00:34:57 UTC
My emailed response didnt make it to the bug for some reason. I removed libvirt and qemu-kvm then reinstalled Ovirt. The errors went away. I would have loved to troubleshoot it to find the root cause but I didnt have time to dig through it. This bug can be closed. Thanks