Bug 1420865 - [downstream clone - 4.0.7] ovirt-engine can't install legacy RHV-H in 3.6 Compatibility Mode
Summary: [downstream clone - 4.0.7] ovirt-engine can't install legacy RHV-H in 3.6 Com...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-host-deploy
Version: 4.0.6
Hardware: x86_64
OS: Linux
high
low
Target Milestone: ovirt-4.0.7
: ---
Assignee: Simone Tiraboschi
QA Contact: Pavol Brilla
URL:
Whiteboard:
Depends On: 1412906
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-02-09 16:48 UTC by rhev-integ
Modified: 2017-03-16 15:40 UTC (History)
17 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Previously, Red Hat Virtualization Manager couldn't install legacy RHEV-H when in 3.6 Compatibility Mode. This is now fixed.
Clone Of: 1412906
Environment:
Last Closed: 2017-03-16 15:40:55 UTC
oVirt Team: Integration
Target Upstream Version:


Attachments (Terms of Use)
engine log of 4.0.7 engine (98.21 KB, text/plain)
2017-03-03 14:07 UTC, Pavol Brilla
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2017:0550 0 normal SHIPPED_LIVE ovirt-host-deploy bug fix update for RHV 4.0.7 2017-03-16 19:27:10 UTC
oVirt gerrit 71608 0 None MERGED kernel: avoid runnning grubby on vintage node 2020-12-23 00:26:26 UTC
oVirt gerrit 71628 0 None MERGED kernel: avoid runnning grubby on vintage node 2020-12-23 00:25:54 UTC
oVirt gerrit 71629 0 None MERGED kernel: avoid runnning grubby on vintage node 2020-12-23 00:25:54 UTC
oVirt gerrit 73607 0 ovirt-host-deploy-1.5 ABANDONED he: Removed obsolete workaround 2020-12-23 00:25:54 UTC
oVirt gerrit 73610 0 ovirt-host-deploy-1.5 MERGED ha: fixing an old workaround 2020-12-23 00:25:52 UTC

Description rhev-integ 2017-02-09 16:48:54 UTC
+++ This bug is a downstream clone. The original bug is: +++
+++   bug 1412906 +++
======================================================================

Description of problem:

Installation of legacy RHV-H in a DC with 3.6 Compatibility Mode fails.

2017-01-13 15:24:43,310 ERROR [org.ovirt.engine.core.uutils.ssh.SSHDialog] (org.ovirt.thread.pool-6-thread-23) [51a24a3b] SSH error running command root@10.64.24.212:'umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -d -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; tar --warning=no-timestamp -C "${MYTMP}" -x &&  "${MYTMP}"/ovirt-host-deploy DIALOG/dialect=str:machine DIALOG/customization=bool:True': Command returned failure code 2 during SSH session 'root@10.64.24.212'


2017-01-13 15:24:43,891 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) [51a24a3b] Correlation ID: 51a24a3b, Call Stack: null, Custom Event ID: -1, Message: Failed to install Host Host-2. Internal error: [Errno 30] Read-only file system: 'os.devnull'.

Version-Release number of selected component (if applicable):
ovirt-engine-4.0.6.3-0.1.el7ev.noarch
rhev-hypervisor7-7.2-20160920.1

How reproducible:
100%

Steps to Reproduce:
1. Try to add new Host to 3.6 Level DC.

Actual results:
Fails

Expected results:
Succeeds

(Originally by Germano Veit Michel)

Comment 4 rhev-integ 2017-02-09 16:49:17 UTC
Can you please provide host-deploy logs or maybe a full sos report?
Just to understand where this is failing.

(Originally by Sandro Bonazzola)

Comment 5 rhev-integ 2017-02-09 16:49:22 UTC
(In reply to Sandro Bonazzola from comment #3)
> Can you please provide host-deploy logs or maybe a full sos report?
> Just to understand where this is failing.

Isn't the host just failing to run those commands in comment #0?

I will reproduce it again and attach the logs shortly.

(Originally by Germano Veit Michel)

Comment 6 rhev-integ 2017-02-09 16:49:28 UTC
Created attachment 1242036 [details]
engine install log

There are not really many logs as the failure if very early.

It's failing to run this in the host, so the deploy does not even start to generate logs.

umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -d -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; tar --warning=no-timestamp -C "${MYTMP}" -x &&  "${MYTMP}"/ovirt-host-deploy DIALOG/dialect=str:machine DIALOG/customization=bool:True

with:

Read-only file system: 'os.devnull'.

https://docs.python.org/3/library/os.html#os.devnull

So is it complaining that /dev/null is read-only? Doesn't make much sense to me.

(Originally by Germano Veit Michel)

Comment 7 rhev-integ 2017-02-09 16:49:33 UTC
Simone, can you please check in this is related to bug #1414265

(Originally by Sandro Bonazzola)

Comment 8 rhev-integ 2017-02-09 16:49:38 UTC
I tried adding an host deployed with rhev-hypervisor7-7.2-20160920.1 on an engine 4.1 on a cluster with cl=3.6 and it failed due to:

2017-02-02 16:07:22 DEBUG otopi.context context._executeMethod:142 method exception
Traceback (most recent call last):
  File "/tmp/ovirt-71Zg8vz4ZC/pythonlib/otopi/context.py", line 132, in _executeMethod
    method['method']()
  File "/tmp/ovirt-71Zg8vz4ZC/otopi-plugins/otopi/core/transaction.py", line 93, in _main_end
    self._mainTransaction.commit()
  File "/tmp/ovirt-71Zg8vz4ZC/pythonlib/otopi/transaction.py", line 148, in commit
    element.commit()
  File "/tmp/ovirt-71Zg8vz4ZC/otopi-plugins/ovirt-host-deploy/kernel/kernel.py", line 80, in commit
    self._parent._build_grubby_command(),
  File "/tmp/ovirt-71Zg8vz4ZC/pythonlib/otopi/plugin.py", line 931, in execute
    command=args[0],
RuntimeError: Command '/sbin/grubby' failed to execute
2017-02-02 16:07:22 ERROR otopi.context context._executeMethod:151 Failed to execute stage 'Transaction commit': Command '/sbin/grubby' failed to execute

(Originally by Simone Tiraboschi)

Comment 9 rhev-integ 2017-02-09 16:49:42 UTC
Created attachment 1247156 [details]
host deploy logs for rhev-h

(Originally by Simone Tiraboschi)

Comment 10 rhev-integ 2017-02-09 16:49:48 UTC
On my RHEV-H host it fails with:
[root@c72he20170202h4 admin]# /sbin/grubby --update-kernel DEFAULT
error opening /boot/grub/grub.cfg for read: No such file or directory

(Originally by Simone Tiraboschi)

Comment 11 rhev-integ 2017-02-09 16:49:53 UTC
Once we fix that, the engine is able to add the host but if fails configuring the management bridge.

The issue is:
2017-02-02 18:43:30,966+01 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler3) [33c9782b] HostName = c72he20170202h4.localdomain
2017-02-02 18:43:30,966+01 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler3) [33c9782b] Failed in 'GetCapabilitiesVDS' method, for vds: 'c72he20170202h4.localdomain'; host: 'c72he20170202h4.localdomain': Required SwitchType is not reported.
2017-02-02 18:43:30,966+01 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler3) [33c9782b] Command 'GetCapabilitiesVDSCommand(HostName = c72he20170202h4.localdomain, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='b5ce63a8-9386-4a2f-8cef-53f41f1bcaa9', vds='Host[c72he20170202h4.localdomain,b5ce63a8-9386-4a2f-8cef-53f41f1bcaa9]'})' execution failed: Required SwitchType is not reported.
2017-02-02 18:43:30,966+01 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [33c9782b] Failure to refresh host 'c72he20170202h4.localdomain' runtime info: Required SwitchType is not reported.
2017-02-02 18:43:30,971+01 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler3) [33c9782b] Failed to refresh VDS , vds = 'c72he20170202h4.localdomain' : 'b5ce63a8-9386-4a2f-8cef-53f41f1bcaa9', error = 'Required SwitchType is not reported.', continuing.
2017-02-02 18:43:30,971+01 ERROR [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler3) [33c9782b] Exception: java.lang.IllegalStateException: Required SwitchType is not reported.
        at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerObjectsBuilder.getSwitchType(VdsBrokerObjectsBuilder.java:2143) [vdsbroker.jar:]
        at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerObjectsBuilder.addHostNetworksAndUpdateInterfaces(VdsBrokerObjectsBuilder.java:1744) [vdsbroker.jar:]
        at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerObjectsBuilder.updateNetworkData(VdsBrokerObjectsBuilder.java:1697) [vdsbroker.jar:]
        at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerObjectsBuilder.updateVDSDynamicData(VdsBrokerObjectsBuilder.java:853) [vdsbroker.jar:]
        at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:17) [vdsbroker.jar:]
        at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:111) [vdsbroker.jar:]
        at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:73) [vdsbroker.jar:]
        at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:33) [dal.jar:]
        at org.ovirt.engine.core.vdsbroker.vdsbroker.DefaultVdsCommandExecutor.execute(DefaultVdsCommandExecutor.java:14) [vdsbroker.jar:]
        at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:]
        at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:674) [vdsbroker.jar:]
        at org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring.beforeFirstRefreshTreatment(HostMonitoring.java:628) [vdsbroker.jar:]
        at org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring.refreshVdsRunTimeInfo(HostMonitoring.java:129) [vdsbroker.jar:]
        at org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring.refresh(HostMonitoring.java:85) [vdsbroker.jar:]
        at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:274) [vdsbroker.jar:]
        at sun.reflect.GeneratedMethodAccessor76.invoke(Unknown Source) [:1.8.0_121]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_121]
        at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_121]
        at org.ovirt.engine.core.utils.timer.JobWrapper.invokeMethod(JobWrapper.java:77) [scheduler.jar:]
        at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:51) [scheduler.jar:]
        at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [rt.jar:1.8.0_121]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_121]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_121]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_121]
        at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_121]

(Originally by Simone Tiraboschi)

Comment 12 rhev-integ 2017-02-09 16:49:58 UTC
Moving to ovirt-host-deploy being the issue caused by a call to grubby.
Dropping dependency on bug #1414265 being not related with the issue.

(Originally by Sandro Bonazzola)

Comment 13 rhev-integ 2017-02-09 16:50:04 UTC
rhev-hypervisor7-7.3-20170118.0.iso coms with vdsm 4.17.37-1.el7ev and, with patch https://gerrit.ovirt.org/#/c/71608/ , the host goes up.

(Originally by Simone Tiraboschi)

Comment 16 Pavol Brilla 2017-03-03 14:06:25 UTC
Clean install engine:
# rpm -q rhevm ovirt-host-deploy
rhevm-4.0.7.4-0.1.el7ev.noarch
ovirt-host-deploy-1.5.4-2.el7ev.noarch


Host installed RHEV-H 7.3-20170118.0.el7ev, from TUI RHEV-M setuped.

Engine - Approve button - failed with message:
Failed to install Host 10-34-62-46.rhev.lab.eng.brq.redhat.com. Internal error: [Errno 30] Read-only file system: 'os.devnull'.


On engine /var/log/ovirt-engine/host-deploy/ is empty.

Attaching /var/log/ovirt-engine/engine.log

Comment 17 Pavol Brilla 2017-03-03 14:07:35 UTC
Created attachment 1259554 [details]
engine log of 4.0.7 engine

Comment 18 Sandro Bonazzola 2017-03-06 07:54:20 UTC
(In reply to Pavol Brilla from comment #16)
> Engine - Approve button - failed with message:
> Failed to install Host 10-34-62-46.rhev.lab.eng.brq.redhat.com. Internal
> error: [Errno 30] Read-only file system: 'os.devnull'.

Looks like a RHEV-H issue, not ovirt-host-deploy.

Comment 19 Simone Tiraboschi 2017-03-06 08:30:50 UTC
(In reply to Pavol Brilla from comment #16)
> Host installed RHEV-H 7.3-20170118.0.el7ev, from TUI RHEV-M setuped.
> 
> Engine - Approve button - failed with message:

Pavol, did you also tried installing the RHEV-H host from the engine as a regular host?

Comment 20 Simone Tiraboschi 2017-03-06 14:39:13 UTC
Reproduced:
[root@rhevh1 dfsfsd]# /tmp/dfsfsd/ovirt-host-deploy DIALOG/dialect=str:machine DIALOG/customization=bool:True
***L:ERROR Internal error: [Errno 30] Read-only file system: 'os.devnull'
[root@rhevh1 dfsfsd]# OTOPI_DEBUG=1 /tmp/dfsfsd/ovirt-host-deploy DIALOG/dialect=str:machine DIALOG/customization=bool:True
Loading plugin group ovirt-host-deploy
Loading plugin ovirt-host-deploy:vmconsole (/tmp/dfsfsd/otopi-plugins/ovirt-host-deploy/vmconsole)
Loading plugin ovirt-host-deploy:vdsmhooks (/tmp/dfsfsd/otopi-plugins/ovirt-host-deploy/vdsmhooks)
Loading plugin ovirt-host-deploy:vdsm (/tmp/dfsfsd/otopi-plugins/ovirt-host-deploy/vdsm)
Loading plugin ovirt-host-deploy:tune (/tmp/dfsfsd/otopi-plugins/ovirt-host-deploy/tune)
Loading plugin ovirt-host-deploy:openstack (/tmp/dfsfsd/otopi-plugins/ovirt-host-deploy/openstack)
Loading plugin ovirt-host-deploy:node (/tmp/dfsfsd/otopi-plugins/ovirt-host-deploy/node)
Loading plugin ovirt-host-deploy:kernel (/tmp/dfsfsd/otopi-plugins/ovirt-host-deploy/kernel)
Loading plugin ovirt-host-deploy:kdump (/tmp/dfsfsd/otopi-plugins/ovirt-host-deploy/kdump)
Loading plugin ovirt-host-deploy:hosted-engine (/tmp/dfsfsd/otopi-plugins/ovirt-host-deploy/hosted-engine)
***L:ERROR Internal error: [Errno 30] Read-only file system: 'os.devnull'
Traceback (most recent call last):
  File "/tmp/dfsfsd/pythonlib/otopi/__main__.py", line 88, in main
    installer.execute()
  File "/tmp/dfsfsd/pythonlib/otopi/main.py", line 153, in execute
    sys.exc_info()[2],
  File "/tmp/dfsfsd/pythonlib/otopi/util.py", line 81, in raiseExceptionInformation
    exec('raise info[1], None, info[2]')
  File "/tmp/dfsfsd/pythonlib/otopi/main.py", line 147, in execute
    self.context.loadPlugins()
  File "/tmp/dfsfsd/pythonlib/otopi/context.py", line 798, in loadPlugins
    self._loadPluginGroups(plugindir, needgroups, loadedgroups)
  File "/tmp/dfsfsd/pythonlib/otopi/context.py", line 113, in _loadPluginGroups
    self._loadPlugins(path, path, groupname)
  File "/tmp/dfsfsd/pythonlib/otopi/context.py", line 70, in _loadPlugins
    self._loadPlugins(base, d, groupname)
  File "/tmp/dfsfsd/pythonlib/otopi/context.py", line 101, in _loadPlugins
    os.path.basename(path),
  File "/tmp/dfsfsd/pythonlib/otopi/util.py", line 105, in loadModule
    mod_desc
  File "/tmp/dfsfsd/otopi-plugins/ovirt-host-deploy/hosted-engine/__init__.py", line 27, in <module>
    from . import configureha
  File "/tmp/dfsfsd/otopi-plugins/ovirt-host-deploy/hosted-engine/configureha.py", line 40, in <module>
    with open('os.devnull', 'wb') as empty_stderr:
PluginLoadException: [Errno 30] Read-only file system: 'os.devnull'

Comment 22 Pavol Brilla 2017-03-09 14:57:42 UTC
With new version of ovirt-host-deploy ( 1.5.5-1.el7ev to be specific ), legacy hypervisor based on 7.3 rhel became available in 3.6 data center

Comment 24 errata-xmlrpc 2017-03-16 15:40:55 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2017-0550.html


Note You need to log in before you can comment on or make changes to this bug.