Bug 1626960 - [el7.6]Network parameters IPv4/route/ovirtmgmt are missing during deploying Hosted-Engine
Summary: [el7.6]Network parameters IPv4/route/ovirtmgmt are missing during deploying H...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: rhev-hypervisor-ng
Version: 4.2.7
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: ovirt-4.2.7
: ---
Assignee: Yuval Turgeman
QA Contact: Wei Wang
URL:
Whiteboard:
: 1630346 (view as bug list)
Depends On:
Blocks: 1630247
TreeView+ depends on / blocked
 
Reported: 2018-09-10 08:02 UTC by Wei Wang
Modified: 2019-04-28 08:36 UTC (History)
17 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1630247 (view as bug list)
Environment:
Last Closed: 2018-11-05 15:00:05 UTC
oVirt Team: Node
Target Upstream Version:


Attachments (Terms of Use)
HE deploy fail logs (1.12 MB, application/x-gzip)
2018-09-10 08:02 UTC, Wei Wang
no flags Details
ks file (277 bytes, text/plain)
2018-09-10 08:02 UTC, Wei Wang
no flags Details


Links
System ID Priority Status Summary Last Updated
Red Hat Bugzilla 1630346 urgent CLOSED Node zero hosted-engine deployment fails with RHVH based on RHEL 7.6 2020-10-14 00:28:05 UTC
Red Hat Product Errata RHSA-2018:3470 None None None 2018-11-05 15:02:34 UTC

Internal Links: 1630346

Description Wei Wang 2018-09-10 08:02:05 UTC
Created attachment 1482029 [details]
HE deploy fail logs

Description of problem:
Network parameters IPv4/route/ovirtmgmt are missing during deploying Hosted-Engine. 
When waiting for host's up on engine, the network parameters are missing.

    IPv4 is missing
    default route is missing
    ovirtmgmt is missing
    ifup/ifdown missing execution authority



Version-Release number of selected component (if applicable):
rhvh-4.2.7.0-0.20180906.0
cockpit-bridge-173-5.el7.x86_64
cockpit-ovirt-dashboard-0.11.33-1.el7ev.noarch
cockpit-ws-173-5.el7.x86_64
cockpit-system-173-5.el7.noarch
cockpit-storaged-172-2.el7.noarch
cockpit-machines-ovirt-172-2.el7.noarch
cockpit-dashboard-172-2.el7.x86_64
cockpit-173-5.el7.x86_64
vdsm-4.20.39-1.el7ev.x86_64

How reproducible:
100%

Steps to Reproduce:
1. Clean install rhvh-4.2.7.0-0.20180906.0 with ks file
2. Login to cockpit WebUI with root account
3. Deploy HE

 
Actual results:
Network parameters IPv4/route/ovirtmgmt are missing when waiting for host's up on engine


Expected results:
Hosted-engine should be deployed successfully.

Additional info:
1. After restart network service, hosted-engine can be redeployed successfully.
2. The issue also occurs during host register to RHVM (vdsm).
3. The issue also occurs when upgrading from 4.2.6 to 4.2.7.

Comment 1 Wei Wang 2018-09-10 08:02:53 UTC
Created attachment 1482030 [details]
ks file

Comment 5 Ryan Barry 2018-09-10 12:48:11 UTC
Additional info #3 looks like a separate but to me, though probably related.

Is the HE issue reproducible on RHEL?

Comment 6 yaowen xu 2018-09-11 10:19:58 UTC
Scenario 1: Upgrade from 4.1-0426 to 4.2-0906, not registered to vsdm, successful; 
Scenario 2: Upgrade from 4.1-0426 to 4.2-0906, registered to vsdm, failed, loss of network.

Comment 7 Wei Wang 2018-09-12 11:10:08 UTC
(In reply to Ryan Barry from comment #5)
> Additional info #3 looks like a separate but to me, though probably related.
> 
> Is the HE issue reproducible on RHEL?

No, this HE issue cannot be reproduced with RHEL7.6

Comment 8 Yuval Turgeman 2018-09-17 06:56:57 UTC
From supervdsm.log, any idea ?

MainProcess|jsonrpc/3::INFO::2018-09-10 12:28:36,657::legacy_switch::223::root::(_add_network) Configuring device ovirtmgmt
MainProcess|jsonrpc/3::DEBUG::2018-09-10 12:28:36,658::cmdutils::150::root::(exec_cmd) /bin/systemctl status NetworkManager (cwd None)
MainProcess|jsonrpc/3::DEBUG::2018-09-10 12:28:36,695::cmdutils::158::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0
MainProcess|jsonrpc/3::DEBUG::2018-09-10 12:28:36,699::cmdutils::150::root::(exec_cmd) /sbin/dhclient '-do-you-support-loading-duid-from-lease-files?' (cwd None)
MainProcess|jsonrpc/3::DEBUG::2018-09-10 12:28:36,728::cmdutils::158::root::(exec_cmd) FAILED: <err> = 'Internet Systems Consortium DHCP Client 4.2.5\nCopyright 2004-2013 Internet Systems Consortium.\nAll rights reserved.\nFor info, please visit https://www.isc.org/software/dhcp/\nUsage: dhclient [-4|-6] [-SNTPI1dvrxi] [-nw] [-p <port>] [-D LL|LLT] \n                [-s server-addr] [-cf config-file] [-lf lease-file]\n                [-pf pid-file] [--no-pid] [-e VAR=val]\n                [-I <dhcp-client-identifier>] [-B]\n                [-H <host-name> | -F <fqdn.fqdn>] [-timeout <timeout>]\n                [-V <vendor-class-identifier>]\n                [-R <request option list>]\n                [-sf script-file] [interface]\n\nThis version of ISC DHCP is based on the release available\non ftp.isc.org.  Features have been added and other changes\nhave been made to the base software release in order to make\nit work better with this distribution.\n\nPlease report for this software via the Red Hat Bugzilla site:\n    http://bugzilla.redhat.com\n\nexiting.\n'; <rc> = 1

Comment 9 Edward Haas 2018-09-17 07:57:58 UTC
(In reply to Yuval Turgeman from comment #8)
> From supervdsm.log, any idea ?
> 
Yes, this is intentional.
We issue the command: /sbin/dhclient '-do-you-support-loading-duid-from-lease-files?'
in order to parse the output and determine dhclient capabilities.

Comment 10 Yuval Turgeman 2018-09-17 19:38:23 UTC
From the supervdsm.log :

---
MainProcess|jsonrpc/3::DEBUG::2018-09-17 19:24:29,939::cmdutils::150::root::(exec_cmd) /sbin/ifdown ovirtmgmt (cwd None)
MainProcess|jsonrpc/3::ERROR::2018-09-17 19:24:29,942::supervdsm_server::100::SuperVdsm.ServerCallback::(wrapper) Error in setupNetworks
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/vdsm/supervdsm_server.py", line 98, in wrapper
    res = func(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/vdsm/network/api.py", line 217, in setupNetworks
    _setup_networks(networks, bondings, options)
  File "/usr/lib/python2.7/site-packages/vdsm/network/api.py", line 238, in _setup_networks
    netswitch.configurator.setup(networks, bondings, options, in_rollback)
  File "/usr/lib/python2.7/site-packages/vdsm/network/netswitch/configurator.py", line 140, in setup
    _setup_legacy(legacy_nets, legacy_bonds, options, in_rollback)
  File "/usr/lib/python2.7/site-packages/vdsm/network/netswitch/configurator.py", line 162, in _setup_legacy
    connectivity.check(options)
  File "/usr/lib/python2.7/site-packages/vdsm/network/configurators/__init__.py", line 57, in __exit__
    leftover = self.rollback()
  File "/usr/lib/python2.7/site-packages/vdsm/network/configurators/ifcfg.py", line 96, in rollback
    self.configApplier.restoreBackups()
  File "/usr/lib/python2.7/site-packages/vdsm/network/configurators/ifcfg.py", line 547, in restoreBackups
    stop_devices(self._backups)
  File "/usr/lib/python2.7/site-packages/vdsm/network/configurators/ifcfg.py", line 826, in stop_devices
    ifdown(dev)
  File "/usr/lib/python2.7/site-packages/vdsm/network/configurators/ifcfg.py", line 910, in ifdown
    rc, _, _ = cmd.exec_sync([EXT_IFDOWN, iface])
  File "/usr/lib/python2.7/site-packages/vdsm/network/cmd.py", line 38, in exec_sync
    retcode, out, err = exec_sync_bytes(cmds)
  File "/usr/lib/python2.7/site-packages/vdsm/common/cmdutils.py", line 154, in exec_cmd
    env=env)
  File "/usr/lib64/python2.7/site-packages/subprocess32.py", line 822, in __init__
    restore_signals, start_new_session)
  File "/usr/lib64/python2.7/site-packages/subprocess32.py", line 1567, in _execute_child
    raise child_exception_type(errno_num, err_msg)
OSError: [Errno 13] Permission denied
---

Looking at the system, it seems that /sbin/if{down,up} are not executable which is caused by NetworkManager

[root@node-22206 vdsm]# ls -l /sbin/ifdown /sbin/ifup
-rw-r--r--. 1 root root 1651 Aug 24 10:23 /sbin/ifdown
-rw-r--r--. 1 root root 5010 Aug 24 10:23 /sbin/ifup
[root@node-22206 vdsm]# rpm -q --dump NetworkManager|grep -E "(ifdown|ifup)"
/usr/libexec/nm-ifdown 155 1536666354 0358d4f8e6134986deca4190251dfad9609516a04f37519705e818f80346bd10 0100755 root root 0 0 0 X
/usr/libexec/nm-ifup 153 1536666354 21574606566d3f4447178ac702e013d4d6d06f182a687ac454fc776fe37afc88 0100755 root root 0 0 0 X
/usr/sbin/ifdown 0 1536666356 e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 0100644 root root 0 0 0 X
/usr/sbin/ifup 0 1536666356 e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 0100644 root root 0 0 0 X
[root@node-22206 vdsm]#

Comment 11 Yuval Turgeman 2018-09-17 20:19:15 UTC
Ok, both NetworkManager and initscripts provide /sbin/if{up,down} and NM tries to handle this with update-alternatives.  rpm -q --triggers NetworkManager shows:

triggerin scriptlet (using /bin/sh) -- initscripts
if [ -f /usr/sbin/ifup -a ! -L /usr/sbin/ifup ]; then
    # initscripts package too old, won't let us set an alternative
    /usr/sbin/update-alternatives --remove ifup /usr/libexec/nm-ifup >/dev/null 2>&1 || :
else
    /usr/sbin/update-alternatives --install /usr/sbin/ifup ifup /usr/libexec/nm-ifup 50 \
        --slave /usr/sbin/ifdown ifdown /usr/libexec/nm-ifdown
fi

The files are not symlinks because they're provided in initscripts, but installed with the permissions from NM (not executable as in comment 10).

I'm not sure which one we should use, but for now, to work around this and use initscripts as we do in 7.5, you need to run `rpm --setperms initscripts` first and add the host after

Comment 12 cshao 2018-09-18 10:50:34 UTC
Test above work around and register to VDSM can succeed.

Test steps:
1. Install redhat-virtualization-host-4.2-20180917.0.el7_6
2. Run `rpm --setperms initscripts`
3. Register to engine

Test result:
RHVH can up.

Comment 13 yaowen xu 2018-09-18 12:28:44 UTC
Test above workaround and register to RHVM(vdsm) can succeed.

1. RHVH 4.1 registered to RHVM 4.2;
2. Upgraded to redhat-virtualization-host-4.2-20180917.0.el7_6;
2. Non Responsive and Network parameters IPv4/route/ovirtmgmt are missing;
3. Run `rpm --setperms initscripts`;
4. Re-register to RHVM 4.2;
5. RHVH can up.

Comment 14 Sandro Bonazzola 2018-09-20 06:53:25 UTC
*** Bug 1630346 has been marked as a duplicate of this bug. ***

Comment 16 Wei Wang 2018-09-26 04:41:01 UTC
Test Verison
rhvh-4.2.7.0-0.20180918.0
cockpit-bridge-173-6.el7.x86_64
cockpit-storaged-172-2.el7.noarch
cockpit-173-6.el7.x86_64
cockpit-ovirt-dashboard-0.11.34-1.el7ev.noarch
cockpit-system-173-6.el7.noarch
cockpit-ws-173-6.el7.x86_64
cockpit-machines-ovirt-172-2.el7.noarch
cockpit-dashboard-172-2.el7.x86_64
NetworkManager-1.12.0-6.el7.x86_64

Test Steps:
1. Clean install rhvh-4.2.7.0-0.20180918.0 with ks file
2. Login to cockpit WebUI with root account
3. Deploy HE

Result:
Hosted-engine deploys successfully.

So the bug is fixed, change the status to "VERIFIED"

Comment 19 errata-xmlrpc 2018-11-05 15:00:05 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:3470


Note You need to log in before you can comment on or make changes to this bug.