Bug 1125178

Created attachment 925990 [details]
logs 12.8.14

Failed to add rhel7 host to rhevm3.4.2

vdsm.log:

Traceback (most recent call last):
  File "/usr/share/vdsm/sampling.py", line 435, in run
    sample = self.sample()
  File "/usr/share/vdsm/sampling.py", line 425, in sample
    hs = HostSample(self._pid)
  File "/usr/share/vdsm/sampling.py", line 209, in __init__
    self.timestamp - os.stat(P_VDSM_CLIENT_LOG).st_mtime <
OSError: [Errno 2] No such file or directory: '/var/run/vdsm/client.log'

Also, it seems that engine do not know how to handle with such an error received from vdsm. Engine.log:


2014-08-12 12:05:26,862 ERROR [org.ovirt.engine.core.utils.ssh.SSHDialog] (org.ovirt.thread.pool-4-thread-49) SSH error running command root.102.11:'umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "c
hmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar --warning=no-timestamp -C "${MYTMP}" -x &&  "${MYTMP}"/setup DIALOG/dialect=str:ma
chine DIALOG/customization=bool:True': java.io.IOException: Command returned failure code 1 during SSH session 'root.102.11'


Nir, should I open a separate but on engine about this issue?

This was tested using:
av11
rhevm-3.4.2-0.1.el6ev.noarch
vdsm-4.14.13-1.el7ev.x86_64

Host with RHEL7 installed
Red Hat Enterprise Linux Server release 7.0 (Maipo)

Attaching full logs from vdsm and engine

(In reply to Elad from comment #2)
> Nir, should I open a separate but on engine about this issue?

Yes, this error is not related in any way to this bug and to the fix. The bug is *not* about any error that prevent adding a host to rhel7, but about failure of multipath reload while vdsm starts.

Returning to ON_QA - if you don't see the multipath error any more, please mark this as VERIFIED.

Don't see the multipath error, but host installation fails with 'Command '/bin/systemctl' failed to execute'

I'll open a bug about the other issue

This bug cannot be verified due to https://bugzilla.redhat.com/show_bug.cgi?id=1129232

Based on comment 4, moving to verified.

please be aware to comment #5, also, QA eng are moving bugs to verified.

Aharon, Elad

How to reproduce:

1. Installing a fresh image
2. Installing vdsm without this fix
3. Running vdsm-tool configure --force
4. Starting vdsm

Vdsm start fail because multipathd reload error

How to verify:

1. Installing a fresh image
2. Installing vdsm including this fix
3. Running vdsm-tool configure --force
4. Starting vdsm

Vdsm should not fail to start because of multipathd reload error. It will fail because of bug 1127877 and bug 1129232.

But failing to add a host to engine cannot block this bug. Keeping this open does not help anyone, we have enough real open bugs to handle.

(In reply to Nir Soffer from comment #8)
> Aharon, Elad
> 
> How to reproduce:
> 
> 1. Installing a fresh image
> 2. Installing vdsm without this fix
> 3. Running vdsm-tool configure --force
> 4. Starting vdsm
> 
> Vdsm start fail because multipathd reload error
> 
> How to verify:
> 
> 1. Installing a fresh image
> 2. Installing vdsm including this fix
> 3. Running vdsm-tool configure --force
> 4. Starting vdsm
> 
> Vdsm should not fail to start because of multipathd reload error. It will
> fail because of bug 1127877 and bug 1129232.
> 
> But failing to add a host to engine cannot block this bug. Keeping this open
> does not help anyone, we have enough real open bugs to handle.

That's exactly the scenario I've executed.
The reason for keeping this bug open is because we don't know if the other issues reported in bug 1127877 and in bug 1129232 that prevents vdsm on host with rhel7 installed to be deployed in rhevm comes before the multipathd reload issue or after it during the installation. We think that there is a possibility that the multipathd phase during installation is blocked by the other issues.

(In reply to Elad from comment #9)
> (In reply to Nir Soffer from comment #8)
> The reason for keeping this bug open is because we don't know if the other
> issues reported in bug 1127877 and in bug 1129232 that prevents vdsm on host
> with rhel7 installed to be deployed in rhevm comes before the multipathd
> reload issue or after it during the installation.

bug 1127877 is revealed by the fix for this bug - I found it when testing
the fix for this bug.

I don't have any idea about 1129232, but from looking in the logs, you can
easily tell if the multipathd reload step was completed successfully, or
vdsm start failed before vdsm tried to reload multipath.

If you cannot tell this from the logs, please upload the test logs.

Indeed, from the logs, we can tell that multipath reload went fine:

MainThread::DEBUG::2014-08-12 12:05:29,224::multipath::169::Storage.Misc.excCmd::(setupMultipath) '/usr/bin/sudo -n /usr/bin/cp /tmp/tmpMKn5f6 /etc/multipath.conf' (cwd None)
MainThread::DEBUG::2014-08-12 12:05:29,242::multipath::169::Storage.Misc.excCmd::(setupMultipath) SUCCESS: <err> = ''; <rc> = 0
MainThread::DEBUG::2014-08-12 12:05:29,244::multipath::175::Storage.Misc.excCmd::(setupMultipath) '/usr/bin/sudo -n /sbin/multipath -F' (cwd None)
MainThread::DEBUG::2014-08-12 12:05:29,263::multipath::175::Storage.Misc.excCmd::(setupMultipath) SUCCESS: <err> = ''; <rc> = 0
MainThread::DEBUG::2014-08-12 12:05:29,264::multipath::178::Storage.Misc.excCmd::(setupMultipath) '/usr/bin/sudo -n /usr/bin/vdsm-tool service-reload multipathd' (cwd None)
MainThread::DEBUG::2014-08-12 12:05:29,594::multipath::178::Storage.Misc.excCmd::(setupMultipath) SUCCESS: <err> = ''; <rc> = 0


Therefore, moving to VERIFIED

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2014-1152.html