Bug 1143820

This shouldn't be a clone, ignore.

Created attachment 938754 [details]
vdsm and ovirt-hosted-engine-setup logs

2014-09-17 19:53:36 DEBUG otopi.plugins.otopi.services.systemd plugin.executeRaw:785 execute: ('/bin/systemctl', 'start', 'libvirtd.service'), executable='None', cwd='None', env=None
2014-09-17 19:53:36 DEBUG otopi.plugins.otopi.services.systemd plugin.executeRaw:803 execute-result: ('/bin/systemctl', 'start', 'libvirtd.service'), rc=1
2014-09-17 19:53:36 DEBUG otopi.plugins.otopi.services.systemd plugin.execute:861 execute-output: ('/bin/systemctl', 'start', 'libvirtd.service') stdout:


2014-09-17 19:53:36 DEBUG otopi.plugins.otopi.services.systemd plugin.execute:866 execute-output: ('/bin/systemctl', 'start', 'libvirtd.service') stderr:
Job for libvirtd.service failed. See 'systemctl status libvirtd.service' and 'journalctl -xn' for details.

2014-09-17 19:53:36 DEBUG otopi.context context._executeMethod:152 method exception
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/otopi/context.py", line 142, in _executeMethod
    method['method']()
  File "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/ovirt-hosted-engine-setup/libvirt/configureqemu.py", line 111, in _misc
    self._restartLibvirt()
  File "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/ovirt-hosted-engine-setup/libvirt/configureqemu.py", line 62, in _restartLibvirt
    state=state,
  File "/usr/share/otopi/plugins/otopi/services/systemd.py", line 138, in state
    'start' if state else 'stop'
  File "/usr/share/otopi/plugins/otopi/services/systemd.py", line 77, in _executeServiceCommand
    raiseOnError=raiseOnError
  File "/usr/lib/python2.7/site-packages/otopi/plugin.py", line 871, in execute
    command=args[0],
RuntimeError: Command '/bin/systemctl' failed to execute

Can you provide:
'systemctl status libvirtd.service' and 'journalctl -xn' for details?

I can only see that when libvirtd failed to restart vdsmd got killed by sigterm.

MainThread::DEBUG::2014-09-17 19:53:36,671::vdsm::58::vds::(sigtermHandler) Received signal 15


Also:
Steps to Reproduce:
(1. yum install ovirt-hosted-engine-setup)
(2. screen hosted-engine --deploy)
3. hosted-engine --deploy

does mean you're running 2 instances of hosted-engine --deploy at the same time?

(In reply to Sandro Bonazzola from comment #4)
> Can you provide:
> 'systemctl status libvirtd.service' and 'journalctl -xn' for details?
# systemctl status libvirtd.service
libvirtd.service - Virtualization daemon
   Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled)
   Active: active (running) since Sun 2014-09-21 07:02:48 IDT; 7min ago
     Docs: man:libvirtd(8)
           http://libvirt.org
 Main PID: 653 (libvirtd)
   CGroup: /system.slice/libvirtd.service
           └─653 /usr/sbin/libvirtd --listen

Sep 21 07:02:48 galactica.tlv.redhat.com libvirtd[653]: Module /usr/lib64/libvirt/connection-driver/libvirt_driver_libxl.so not accessible
Sep 21 07:02:48 galactica.tlv.redhat.com libvirtd[653]: Module /usr/lib64/libvirt/connection-driver/libvirt_driver_lxc.so not accessible
Sep 21 07:02:48 galactica.tlv.redhat.com libvirtd[653]: Module /usr/lib64/libvirt/connection-driver/libvirt_driver_uml.so not accessible
Sep 21 07:02:48 galactica.tlv.redhat.com libvirtd[653]: Module /usr/lib64/libvirt/connection-driver/libvirt_driver_vbox.so not accessible
Sep 21 07:02:48 galactica.tlv.redhat.com libvirtd[653]: Unable to lookup SELinux process context: Invalid argument
Sep 21 07:02:48 galactica.tlv.redhat.com systemd[1]: Started Virtualization daemon.
Sep 21 07:02:48 galactica.tlv.redhat.com libvirtd[653]: Unable to lookup SELinux process context: Invalid argument
Sep 21 07:02:48 galactica.tlv.redhat.com libvirtd[653]: Unable to lookup SELinux process context: Invalid argument
Sep 21 07:02:55 galactica.tlv.redhat.com libvirtd[653]: Unable to lookup SELinux process context: Invalid argument
Sep 21 07:02:55 galactica.tlv.redhat.com libvirtd[653]: Unable to lookup SELinux process context: Invalid argument

------------------------------------------------------------------------------

# journalctl -xn
-- Logs begin at Tue 2014-07-15 17:18:07 IDT, end at Sun 2014-09-21 07:10:12 IDT. --
Sep 21 07:10:02 galactica.tlv.redhat.com systemd[1694]: Startup finished in 136ms.
-- Subject: System start-up is now complete
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- All system services necessary queued for starting at boot have been
-- successfully started. Note that this does not mean that the machine is
-- now idle as services might still be busy with completing start-up.
-- 
-- Kernel start-up required KERNEL_USEC microseconds.
-- 
-- Initial RAM disk start-up required INITRD_USEC microseconds.
-- 
-- Userspace start-up required 136770 microseconds.
Sep 21 07:10:02 galactica.tlv.redhat.com systemd[1]: Started User Manager for 0.
-- Subject: Unit user has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit user has finished starting up.
-- 
-- The start-up result is done.
Sep 21 07:10:07 galactica.tlv.redhat.com sshd[1701]: Invalid user vvolansk from 10.35.0.134
Sep 21 07:10:07 galactica.tlv.redhat.com sshd[1701]: input_userauth_request: invalid user vvolansk [preauth]
Sep 21 07:10:09 galactica.tlv.redhat.com sshd[1701]: Connection closed by 10.35.0.134 [preauth]
Sep 21 07:10:12 galactica.tlv.redhat.com sshd[1705]: Accepted publickey for root from 10.35.0.134 port 34644 ssh2: RSA ab:a9:1e:2d:cd:ff:e8:1d:c2:00:c6:e0:24:93:a0:ef
Sep 21 07:10:12 galactica.tlv.redhat.com systemd[1]: Starting Session 2 of user root.
-- Subject: Unit session-2.scope has begun with start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit session-2.scope has begun starting up.
Sep 21 07:10:12 galactica.tlv.redhat.com systemd-logind[642]: New session 2 of user root.
-- Subject: A new session 2 has been created for user root
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- Documentation: http://www.freedesktop.org/wiki/Software/systemd/multiseat
-- 
-- A new session with the ID 2 has been created for the user root.
-- 
-- The leading process of the session is 1705.
> 
> I can only see that when libvirtd failed to restart vdsmd got killed by
> sigterm.
> 
> MainThread::DEBUG::2014-09-17 19:53:36,671::vdsm::58::vds::(sigtermHandler)
> Received signal 15
> 
> 
> Also:
> Steps to Reproduce:
> (1. yum install ovirt-hosted-engine-setup)
> (2. screen hosted-engine --deploy)
> 3. hosted-engine --deploy
> 
> does mean you're running 2 instances of hosted-engine --deploy at the same
> time?

No, I mean this is what happened after bz1142707, in which screen was run and deploy failed. I added it just in case there was any residue.
hosted-engine --deploy ran alone.

Hi Vered,
Can you try the same "hosted-engine --deploy" for 30 Gig LUN?
As a prerequisites, hosted-engine requires a bit more than 20Gig storage space:
http://community.redhat.com/blog/2014/05/ovirt-3-4-glusterized/

"The hosted engine VM will require 20GB, and you'll want to have plenty of storage space left over for the VMs you'll create and manage with oVirt."

libvirt might be stuck as a result of insufficient space on LUN.

(In reply to Nikolai Sednev from comment #6)
> Hi Vered,
> Can you try the same "hosted-engine --deploy" for 30 Gig LUN?
> As a prerequisites, hosted-engine requires a bit more than 20Gig storage
> space:
> http://community.redhat.com/blog/2014/05/ovirt-3-4-glusterized/
> 
> "The hosted engine VM will require 20GB, and you'll want to have plenty of
> storage space left over for the VMs you'll create and manage with oVirt."
> 
> libvirt might be stuck as a result of insufficient space on LUN.

I don't have the resources to test this now, sorry...
If 30GB works for you feel free to close (though I still think an appropriate message should be displayed if this is indeed the cause in 20GB).

Simone, hasn't this been fixed for 3.5.2? IIRC we now check the LUN size against the overhead required by VDSM.

Yes, it's basically a duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=1142710
Now we consider the storage domain overhead having an explicit check against the free space in the VG after the storage domain creation.

*** This bug has been marked as a duplicate of bug 1142710 ***