Bug 1179068

Summary: [3.5-6.6] libvirtd service is not active as default when RHEV-H TUI installation
Product: Red Hat Enterprise Virtualization Manager Reporter: cshao <cshao>
Component: ovirt-nodeAssignee: Douglas Schilling Landgraf <dougsland>
Status: CLOSED ERRATA QA Contact: Virtualization Bugs <virt-bugs>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 3.5.0CC: cshao, danken, dougsland, ecohen, fdeutsch, gklein, hadong, huiwa, iheim, lsurette, rbarry, yaniwang, ycui, ylavi
Target Milestone: ---Keywords: Regression
Target Release: 3.5.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: node
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-02-11 21:07:46 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Node RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1164308, 1164311    
Attachments:
Description Flags
libvirt-issue.png
none
libvirt.tar.gz none

Description cshao 2015-01-06 06:27:24 UTC
Created attachment 976714 [details]
libvirt-issue.png

Description of problem:
libvirtd service is not active as default when RHEV-H TUI installation

Version-Release number of selected component (if applicable):
rhev-hypervisor6-6.6-20150105.0
ovirt-node-3.1.0-0.39.20150105gitb784105.el6.noarch
vdsm-4.16.8.1-4.el6ev.x86_64
libvirt-0.10.2-46.el6_6.2.x86_64

How reproducible:
100%

Steps to Reproduce:
1. Boot from PXE, TUI install RHEV-H
2. Focus on the first installation screen and check the libvirt error.
3. Finish the installation and log in the hypervisor.
4. Check libvirt issue on TUI.
5. Please see attachment for more details.

Actual results:
1. Step 2: "Failed to Establish Libvirt Connection" appears in the first installation screen.
2. Step4: "Failed to Establish Libvirt Connection" still appears on TUI.
3. Can manually start libvirtd service successful.

=============================
# /etc/init.d/libvirtd status
libvirtd is stopped

Expected results:
libvirtd service can work fine.

Additional info:
1. No such issue with rhev-hypervisor6-6.6-20141218.0, so it is a regression bug.
2. Automatic installation did not meet this issue, after automation installation, the libvirtd is running as default.
3. No such issue on rhev-hypervisor7-7.0-20150105.0 build.

Comment 1 cshao 2015-01-06 06:29:03 UTC
Created attachment 976715 [details]
libvirt.tar.gz

Comment 2 Douglas Schilling Landgraf 2015-01-06 21:26:12 UTC
Hi,

I have reproduced the report. I don't see the libvirtd daemon after installation.

VDSM init script contain a trigger to start libvirtd and it's not working because the others needed service are not starting as expected and it cannot continue on the loop to start libvirtd.

Some data:
===============
init/sysvinit/vdsmd.init.in
<snip>
NEEDED_SERVICES="multipathd rpcbind ntpd wdmd sanlock network libvirtd supervdsmd"

start() {
   <snip>
   start_needed_srv "${NEEDED_SERVICES}" || return 1
<snip>

start_needed_srv() {
    local srv
    local ret_val
    local needed_services="$1"

    for srv in ${needed_services}; do
        if initctl status "${srv}" >/dev/null 2>&1; then
            # When srv is Upstart service, status srv always returns 0
            initctl start "${srv}" || : # start fails when already running
            initctl status "${srv}" | grep -q start/running
        else
            service "${srv}" status >/dev/null 2>&1 || service "${srv}" start
        fi
        ret_val=$?
        if [ "${ret_val}" -ne 0 ]; then
            log_failure_msg "${prog}: Start dependent ${srv}"
            return "${ret_val}"
        fi
    done
}

So, if we add a debugging in the for:

    for srv in ${needed_services}; do
        echo ${srv} >> /tmp/starting
        ....
        ret_val=$?
        echo ${ret_val} >> /tmp/starting

We will see that vdsm cannot go into the all services if one stop.

# service vdsmd start
# cat /tmp/starting
multipathd
0
rpcbind
6

However, if we change the order of NEEDED_SERVICES to libvirtd be the first service to be started and reboot, everything works as expected.

Other data about the host
==================================
In case, change the network for the first service, I saw:
# service network start
# echo $?
6

# ifconfig
lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:24993494 errors:0 dropped:0 overruns:0 frame:0
          TX packets:24993494 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:28158 (27.4 KiB)  TX bytes:28158 (27.4 KiB)

One option is make the libvirtd service be started on the boot without intervention of VDSM.

Comment 3 Douglas Schilling Landgraf 2015-01-06 21:38:45 UTC
I don't see it happening in RHEV-H7 because even we having in rhevh7-install.ks --disabled for libvirtd:

=====
services --enabled=auditd,ntpd,ntpdate,iptables,network,rsyslog,multipathd,snmpd,ovirt-early,ovirt-post,ovirt-cim,ovirt-kdump,cgconfig,mcelog,tuned --disabled=netfs,ovirt-awake,libvirt-guests,libvirtd,kdump
====

We do also have in post rhevh7-post.ks:

========
#enabling libvirtd as described in its libvirtd.service comments and virt-who as requested in bug
systemctl enable libvirtd.service
systemctl enable virt-who.service
========

Comment 4 Douglas Schilling Landgraf 2015-01-07 13:09:10 UTC
*** Bug 1179148 has been marked as a duplicate of this bug. ***

Comment 10 cshao 2015-01-19 05:47:04 UTC
Test version:
rhev-hypervisor6-6.6-20150114.0
ovirt-node-3.2.1-4.el6.noarch

# /etc/init.d/libvirtd status
libvirtd (pid  16657) is running...

Test result:
Libvirtd service is active as default when RHEV-H TUI installation.
So the bug has been fixed, change bug status to VERIFIED.

Comment 12 errata-xmlrpc 2015-02-11 21:07:46 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2015-0160.html