Bug 1143820 - libvirtd failed to start while deploying hosted-engine
Summary: libvirtd failed to start while deploying hosted-engine
Keywords:
Status: CLOSED DUPLICATE of bug 1142710
Alias: None
Product: oVirt
Classification: Retired
Component: ovirt-hosted-engine-setup
Version: 3.5
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 3.5.2
Assignee: Simone Tiraboschi
QA Contact: Nikolai Sednev
URL:
Whiteboard: integration
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-09-18 05:25 UTC by Vered Volansky
Modified: 2016-05-05 04:38 UTC (History)
12 users (show)

Fixed In Version:
Clone Of: 1142707
Environment:
Last Closed: 2015-02-18 14:38:24 UTC
oVirt Team: ---
Embargoed:


Attachments (Terms of Use)
vdsm and ovirt-hosted-engine-setup logs (1.46 MB, application/octet-stream)
2014-09-18 05:55 UTC, Vered Volansky
no flags Details

Description Vered Volansky 2014-09-18 05:25:06 UTC
Description of problem:
yum install ovirt-hosted-engine-setup was previously done, this was a second attempt to deploy, after bz1142707.
hosted-engine --deploy gets stuck until a broken pipe (used ssh to the host).

Version-Release number of selected component (if applicable):
ovirt-hosted-engine-setup-1.2.0-0.1.master.20140908073913.git480e272.fc20.noarch

How reproducible:
Happened 1/2.

Steps to Reproduce:
(1. yum install ovirt-hosted-engine-setup)
(2. screen hosted-engine --deploy)
3. hosted-engine --deploy

Went through the entire process.
Chose iscsi target, anything that COULD be default WAS default other than iscsi and boot (set to PXE).
Verified the settings I chose.

Actual results:
got stuck on Configuring libvirt .

Expected results:
Finish process successfully :)

Additional info:
I used a 20G brand new lun.

Comment 1 Vered Volansky 2014-09-18 05:26:23 UTC
This shouldn't be a clone, ignore.

Comment 2 Vered Volansky 2014-09-18 05:55:52 UTC
Created attachment 938754 [details]
vdsm and ovirt-hosted-engine-setup logs

Comment 3 Sandro Bonazzola 2014-09-18 11:00:53 UTC
2014-09-17 19:53:36 DEBUG otopi.plugins.otopi.services.systemd plugin.executeRaw:785 execute: ('/bin/systemctl', 'start', 'libvirtd.service'), executable='None', cwd='None', env=None
2014-09-17 19:53:36 DEBUG otopi.plugins.otopi.services.systemd plugin.executeRaw:803 execute-result: ('/bin/systemctl', 'start', 'libvirtd.service'), rc=1
2014-09-17 19:53:36 DEBUG otopi.plugins.otopi.services.systemd plugin.execute:861 execute-output: ('/bin/systemctl', 'start', 'libvirtd.service') stdout:


2014-09-17 19:53:36 DEBUG otopi.plugins.otopi.services.systemd plugin.execute:866 execute-output: ('/bin/systemctl', 'start', 'libvirtd.service') stderr:
Job for libvirtd.service failed. See 'systemctl status libvirtd.service' and 'journalctl -xn' for details.

2014-09-17 19:53:36 DEBUG otopi.context context._executeMethod:152 method exception
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/otopi/context.py", line 142, in _executeMethod
    method['method']()
  File "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/ovirt-hosted-engine-setup/libvirt/configureqemu.py", line 111, in _misc
    self._restartLibvirt()
  File "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/ovirt-hosted-engine-setup/libvirt/configureqemu.py", line 62, in _restartLibvirt
    state=state,
  File "/usr/share/otopi/plugins/otopi/services/systemd.py", line 138, in state
    'start' if state else 'stop'
  File "/usr/share/otopi/plugins/otopi/services/systemd.py", line 77, in _executeServiceCommand
    raiseOnError=raiseOnError
  File "/usr/lib/python2.7/site-packages/otopi/plugin.py", line 871, in execute
    command=args[0],
RuntimeError: Command '/bin/systemctl' failed to execute

Comment 4 Sandro Bonazzola 2014-09-18 11:04:43 UTC
Can you provide:
'systemctl status libvirtd.service' and 'journalctl -xn' for details?

I can only see that when libvirtd failed to restart vdsmd got killed by sigterm.

MainThread::DEBUG::2014-09-17 19:53:36,671::vdsm::58::vds::(sigtermHandler) Received signal 15


Also:
Steps to Reproduce:
(1. yum install ovirt-hosted-engine-setup)
(2. screen hosted-engine --deploy)
3. hosted-engine --deploy

does mean you're running 2 instances of hosted-engine --deploy at the same time?

Comment 5 Vered Volansky 2014-09-21 04:10:04 UTC
(In reply to Sandro Bonazzola from comment #4)
> Can you provide:
> 'systemctl status libvirtd.service' and 'journalctl -xn' for details?
# systemctl status libvirtd.service
libvirtd.service - Virtualization daemon
   Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled)
   Active: active (running) since Sun 2014-09-21 07:02:48 IDT; 7min ago
     Docs: man:libvirtd(8)
           http://libvirt.org
 Main PID: 653 (libvirtd)
   CGroup: /system.slice/libvirtd.service
           └─653 /usr/sbin/libvirtd --listen

Sep 21 07:02:48 galactica.tlv.redhat.com libvirtd[653]: Module /usr/lib64/libvirt/connection-driver/libvirt_driver_libxl.so not accessible
Sep 21 07:02:48 galactica.tlv.redhat.com libvirtd[653]: Module /usr/lib64/libvirt/connection-driver/libvirt_driver_lxc.so not accessible
Sep 21 07:02:48 galactica.tlv.redhat.com libvirtd[653]: Module /usr/lib64/libvirt/connection-driver/libvirt_driver_uml.so not accessible
Sep 21 07:02:48 galactica.tlv.redhat.com libvirtd[653]: Module /usr/lib64/libvirt/connection-driver/libvirt_driver_vbox.so not accessible
Sep 21 07:02:48 galactica.tlv.redhat.com libvirtd[653]: Unable to lookup SELinux process context: Invalid argument
Sep 21 07:02:48 galactica.tlv.redhat.com systemd[1]: Started Virtualization daemon.
Sep 21 07:02:48 galactica.tlv.redhat.com libvirtd[653]: Unable to lookup SELinux process context: Invalid argument
Sep 21 07:02:48 galactica.tlv.redhat.com libvirtd[653]: Unable to lookup SELinux process context: Invalid argument
Sep 21 07:02:55 galactica.tlv.redhat.com libvirtd[653]: Unable to lookup SELinux process context: Invalid argument
Sep 21 07:02:55 galactica.tlv.redhat.com libvirtd[653]: Unable to lookup SELinux process context: Invalid argument

------------------------------------------------------------------------------

# journalctl -xn
-- Logs begin at Tue 2014-07-15 17:18:07 IDT, end at Sun 2014-09-21 07:10:12 IDT. --
Sep 21 07:10:02 galactica.tlv.redhat.com systemd[1694]: Startup finished in 136ms.
-- Subject: System start-up is now complete
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- All system services necessary queued for starting at boot have been
-- successfully started. Note that this does not mean that the machine is
-- now idle as services might still be busy with completing start-up.
-- 
-- Kernel start-up required KERNEL_USEC microseconds.
-- 
-- Initial RAM disk start-up required INITRD_USEC microseconds.
-- 
-- Userspace start-up required 136770 microseconds.
Sep 21 07:10:02 galactica.tlv.redhat.com systemd[1]: Started User Manager for 0.
-- Subject: Unit user has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit user has finished starting up.
-- 
-- The start-up result is done.
Sep 21 07:10:07 galactica.tlv.redhat.com sshd[1701]: Invalid user vvolansk from 10.35.0.134
Sep 21 07:10:07 galactica.tlv.redhat.com sshd[1701]: input_userauth_request: invalid user vvolansk [preauth]
Sep 21 07:10:09 galactica.tlv.redhat.com sshd[1701]: Connection closed by 10.35.0.134 [preauth]
Sep 21 07:10:12 galactica.tlv.redhat.com sshd[1705]: Accepted publickey for root from 10.35.0.134 port 34644 ssh2: RSA ab:a9:1e:2d:cd:ff:e8:1d:c2:00:c6:e0:24:93:a0:ef
Sep 21 07:10:12 galactica.tlv.redhat.com systemd[1]: Starting Session 2 of user root.
-- Subject: Unit session-2.scope has begun with start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit session-2.scope has begun starting up.
Sep 21 07:10:12 galactica.tlv.redhat.com systemd-logind[642]: New session 2 of user root.
-- Subject: A new session 2 has been created for user root
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- Documentation: http://www.freedesktop.org/wiki/Software/systemd/multiseat
-- 
-- A new session with the ID 2 has been created for the user root.
-- 
-- The leading process of the session is 1705.
> 
> I can only see that when libvirtd failed to restart vdsmd got killed by
> sigterm.
> 
> MainThread::DEBUG::2014-09-17 19:53:36,671::vdsm::58::vds::(sigtermHandler)
> Received signal 15
> 
> 
> Also:
> Steps to Reproduce:
> (1. yum install ovirt-hosted-engine-setup)
> (2. screen hosted-engine --deploy)
> 3. hosted-engine --deploy
> 
> does mean you're running 2 instances of hosted-engine --deploy at the same
> time?

No, I mean this is what happened after bz1142707, in which screen was run and deploy failed. I added it just in case there was any residue.
hosted-engine --deploy ran alone.

Comment 6 Nikolai Sednev 2015-02-15 17:14:29 UTC
Hi Vered,
Can you try the same "hosted-engine --deploy" for 30 Gig LUN?
As a prerequisites, hosted-engine requires a bit more than 20Gig storage space:
http://community.redhat.com/blog/2014/05/ovirt-3-4-glusterized/

"The hosted engine VM will require 20GB, and you'll want to have plenty of storage space left over for the VMs you'll create and manage with oVirt."

libvirt might be stuck as a result of insufficient space on LUN.

Comment 7 Vered Volansky 2015-02-16 11:06:21 UTC
(In reply to Nikolai Sednev from comment #6)
> Hi Vered,
> Can you try the same "hosted-engine --deploy" for 30 Gig LUN?
> As a prerequisites, hosted-engine requires a bit more than 20Gig storage
> space:
> http://community.redhat.com/blog/2014/05/ovirt-3-4-glusterized/
> 
> "The hosted engine VM will require 20GB, and you'll want to have plenty of
> storage space left over for the VMs you'll create and manage with oVirt."
> 
> libvirt might be stuck as a result of insufficient space on LUN.

I don't have the resources to test this now, sorry...
If 30GB works for you feel free to close (though I still think an appropriate message should be displayed if this is indeed the cause in 20GB).

Comment 8 Sandro Bonazzola 2015-02-18 14:26:24 UTC
Simone, hasn't this been fixed for 3.5.2? IIRC we now check the LUN size against the overhead required by VDSM.

Comment 9 Simone Tiraboschi 2015-02-18 14:38:24 UTC
Yes, it's basically a duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=1142710
Now we consider the storage domain overhead having an explicit check against the free space in the VG after the storage domain creation.

*** This bug has been marked as a duplicate of bug 1142710 ***


Note You need to log in before you can comment on or make changes to this bug.