Bug 1318812

Dan, can you please check why the host is failing network configuration if nrpe is installed? I guess people may be interested in monitoring network with Nagios so I'd like to avoid to remove it during the host deployment.

Darryl, please attach {super,}vdsm.log from the host that you fail to install.

Created attachment 1139313 [details]
vdsm.log

Log of first failed attempt

Created attachment 1139314 [details]
supervdsm.log

I just removed and installed a host that was otherwise working fine. It had it's networks configured etc. It successfully activated with nrpe running.

not necessarily related, but we see the ping tightloop solved by https://gerrit.ovirt.org/#/c/54644/

08:59:51,535::__init__::533::jsonrpc.JsonRpcServer::(_serveRequest) Return 'Host.ping' in bridge with True
jsonrpc.Executor/6::DEBUG::2016-03-17 08:59:51,538::__init__::503::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'Host.ping' in bridge with {}
jsonrpc.Executor/6::DEBUG::2016-03-17 08:59:51,538::__init__::533::jsonrpc.JsonRpcServer::(_serveRequest) Return 'Host.ping' in bridge with True
jsonrpc.Executor/7::DEBUG::2016-03-17 08:59:51,707::__init__::503::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'Host.getCapabilities' in bridge with {}



'enp9s0': {'addr': '10.4.14.42', 'ipv6gateway': '::', 'ipv6addrs': ['fe80::76d0:2bff:fec8:dcba/64'], 'mtu': '1500', 'dhcpv4': False, 'netmask': '255.255.0.0', 'dhcpv6': False, 'ipv4addrs': ['10.4.14.42/16'], 'cfg': {'DOMAIN': 'gps.local', 'NAME': 'enp9s0', 'DNS3': '10.4.171.3', 'DNS2': '10.4.171.2', 'DNS1': '10.4.171.1', 'DEFROUTE': 'yes', 'IPV6_AUTOCONF': 'yes', 'PREFIX': '16', 'IPV6_DEFROUTE': 'yes', 'IPV6_FAILURE_FATAL': 'no', 'IPV6_PEERROUTES': 'yes', 'IPV4_FAILURE_FATAL': 'no', 'ONBOOT': 'yes', 'IPV6_PEERDNS': 'yes', 'IPV6_PRIVACY': 'no', 'BOOTPROTO': 'none', 'DEVICE': 'enp9s0', 'UUID': 'de4f2cce-e2dd-452a-bf6a-ba48d69d4344', 'IPV6INIT': 'yes', 'IPADDR': '10.4.14.42', 'TYPE': 'Ethernet', 'GATEWAY': '10.4.254.254'}, 'hwaddr': '74:d0:2b:c8:dc:ba', 'speed': 1000, 'gateway': '10.4.254.254'}

did you add the host with 10.4.14.42 addr?

The vdsm and supervdsm logs seem to be from different time ranges.
Please recreate the problem and send the two logs of that period.

Created attachment 1139805 [details]
host h5 vdsm log

VDSM.log on failed host

Created attachment 1139806 [details]
supervdsm log  on h5

supoervdsm log on failed host

Here is a freshly installed host with nrpe installed which fails in th esame way as before.

Note: the networks must be configured on the host after it is installed.
The first thing you notice is that ovirtmgmt network is not attached to the initial interface. When NRPE is removed, this succeeds.

Created attachment 1139810 [details]
ovirt-host-deploy log from failed deploy

Created attachment 1139812 [details]
vdsm.log after successful re-install after nrpe is removed

Created attachment 1139813 [details]
supervdsm.log after successful deploy after nrpe was removed

Created attachment 1139815 [details]
engine log after successful deploy

Performed a re-install after nrpe package was removed.
1. Could not be performed until the failed network tasks were removed. The only way I know to do this is to run engine-setup on the engine which clears tasks and locks.

2. Install still fails due to the network not being configured correctly. Some of the storage relies on the additional networks having IP addresses on the host.

3. The ovirtmgmt network is attached to the default interface after nrpe is removed and the networks can be configured as normal.

4. The host is then successfully activated

Further information:
If the host (after the initial successful install with nrpe removed) has nrpe re-installed and started and is then is removed and re-added (via New) it will activate correctly. I assume due to the networks being set up ready to go. Nrpe somehow affects the attachment of ovirtmgmt to the default interface which then breaks the network configuration.

Comment on attachment 1139810 [details]
ovirt-host-deploy log from failed deploy

Can you please provide the engile.log file from the Engine server?
var/log/ovirt-engine/engine.log

Created attachment 1141050 [details]
Engine log for failed deploy

Created attachment 1141062 [details]
Engine.log after successful deploy

Moving from 4.0 alpha to 4.0 beta since 4.0 alpha has been already released and bug is not ON_QA.

oVirt 4.0 beta has been released, moving to RC milestone.

oVirt 4.0 beta has been released, moving to RC milestone.

I could not see any useful information from Engine or VDSM, with relation to networking.
Recreating a setup with nrpe installed has not shown any problem.


I can see an error in the ovirt-host-deploy log (see below).
Didi, is this a known issue? Is it related?

2016-03-24 10:48:58 DEBUG otopi.plugins.otopi.packagers.dnfpackager dnfpackager._boot:178 Cannot initialize minidnf
Traceback (most recent call last):
  File "/tmp/ovirt-8HvEFjNVVJ/otopi-plugins/otopi/packagers/dnfpackager.py", line 165, in _boot
    constants.PackEnv.DNF_DISABLED_PLUGINS
  File "/tmp/ovirt-8HvEFjNVVJ/otopi-plugins/otopi/packagers/dnfpackager.py", line 75, in _getMiniDNF
    from otopi import minidnf
  File "/tmp/ovirt-8HvEFjNVVJ/pythonlib/otopi/minidnf.py", line 31, in <module>
    import dnf
ImportError: No module named dnf
2016-03-24 10:48:58 DEBUG otopi.context context.dumpEnvironment:500 ENVIRONMENT DUMP - BEGIN

(In reply to Edward Haas from comment #22)
> I could not see any useful information from Engine or VDSM, with relation to
> networking.
> Recreating a setup with nrpe installed has not shown any problem.
> 
> 
> I can see an error in the ovirt-host-deploy log (see below).
> Didi, is this a known issue? Is it related?
> 
> 2016-03-24 10:48:58 DEBUG otopi.plugins.otopi.packagers.dnfpackager
> dnfpackager._boot:178 Cannot initialize minidnf
> Traceback (most recent call last):
>   File "/tmp/ovirt-8HvEFjNVVJ/otopi-plugins/otopi/packagers/dnfpackager.py",
> line 165, in _boot
>     constants.PackEnv.DNF_DISABLED_PLUGINS
>   File "/tmp/ovirt-8HvEFjNVVJ/otopi-plugins/otopi/packagers/dnfpackager.py",
> line 75, in _getMiniDNF
>     from otopi import minidnf
>   File "/tmp/ovirt-8HvEFjNVVJ/pythonlib/otopi/minidnf.py", line 31, in
> <module>
>     import dnf
> ImportError: No module named dnf
> 2016-03-24 10:48:58 DEBUG otopi.context context.dumpEnvironment:500
> ENVIRONMENT DUMP - BEGIN

No, it's unrelated and can be ignored.

Normally, since we added dnf support, either it or yum will fail, depending on which is installed.

If there is a real problem with the packager, it will fail later on when actually used.

Daryl, do you - unlike Edy - still see the bug? Could you somehow let us log into your system to understand what is wrong there?

I just re-installed one of the hosts from scratch. Still exhibits the same fault.
I just noticed the journal refers to a failure to configure network vdsm-ovirtmgt. The ovirtmgt network is configured.
Jun 14 13:14:58 ovirt36-h7.gps.local multipathd[20928]: dm-3: remove map (uevent)
Jun 14 13:14:58 ovirt36-h7.gps.local multipathd[20928]: dm-3: remove map (uevent)
Jun 14 13:15:01 ovirt36-h7.gps.local vdsm[21130]: vdsm vds.dispatcher ERROR SSL error during reading data: unexpected eof
Jun 14 13:15:01 ovirt36-h7.gps.local python[20910]: DIGEST-MD5 client step 2
Jun 14 13:15:01 ovirt36-h7.gps.local python[20910]: DIGEST-MD5 parse_server_challenge()
Jun 14 13:15:01 ovirt36-h7.gps.local python[20910]: DIGEST-MD5 ask_user_info()
Jun 14 13:15:01 ovirt36-h7.gps.local python[20910]: DIGEST-MD5 client step 2
Jun 14 13:15:01 ovirt36-h7.gps.local python[20910]: DIGEST-MD5 ask_user_info()
Jun 14 13:15:01 ovirt36-h7.gps.local python[20910]: DIGEST-MD5 make_client_response()
Jun 14 13:15:01 ovirt36-h7.gps.local python[20910]: DIGEST-MD5 client step 3
Jun 14 13:15:02 ovirt36-h7.gps.local systemd[1]: Started /usr/sbin/ifup enp9s0.
Jun 14 13:15:02 ovirt36-h7.gps.local systemd[1]: Starting /usr/sbin/ifup enp9s0.
Jun 14 13:15:02 ovirt36-h7.gps.local kernel: r8169 0000:09:00.0 enp9s0: link down
Jun 14 13:15:02 ovirt36-h7.gps.local kernel: IPv6: ADDRCONF(NETDEV_UP): enp9s0: link is not ready
Jun 14 13:15:02 ovirt36-h7.gps.local kernel: r8169 0000:09:00.0 enp9s0: link down
Jun 14 13:15:02 ovirt36-h7.gps.local kernel: device enp9s0 entered promiscuous mode
Jun 14 13:15:02 ovirt36-h7.gps.local systemd[1]: Started /usr/sbin/ifup ovirtmgmt.
Jun 14 13:15:02 ovirt36-h7.gps.local systemd[1]: Starting /usr/sbin/ifup ovirtmgmt.
Jun 14 13:15:02 ovirt36-h7.gps.local kernel: IPv6: ADDRCONF(NETDEV_UP): ovirtmgmt: link is not ready
Jun 14 13:15:04 ovirt36-h7.gps.local daemonAdapter[20910]: libvirt: Network Driver error : Network not found: no network with matching name 'vdsm-ovirtmgmt'
Jun 14 13:15:05 ovirt36-h7.gps.local kernel: r8169 0000:09:00.0 enp9s0: link up
Jun 14 13:15:05 ovirt36-h7.gps.local kernel: IPv6: ADDRCONF(NETDEV_CHANGE): enp9s0: link becomes ready
Jun 14 13:15:05 ovirt36-h7.gps.local kernel: ovirtmgmt: port 1(enp9s0) entered forwarding state
Jun 14 13:15:05 ovirt36-h7.gps.local kernel: ovirtmgmt: port 1(enp9s0) entered forwarding state
Jun 14 13:15:05 ovirt36-h7.gps.local kernel: IPv6: ADDRCONF(NETDEV_CHANGE): ovirtmgmt: link becomes ready
Jun 14 13:15:07 ovirt36-h7.gps.local vdsm[21130]: vdsm vds.dispatcher ERROR SSL error during reading data: unexpected eof
Jun 14 13:15:09 ovirt36-h7.gps.local sshd[18567]: pam_unix(sshd:session): session closed for user root
Jun 14 13:15:09 ovirt36-h7.gps.local systemd-logind[823]: Removed session 9.

Please contact me directly via email to arrange a log in if necessary.

Oops, it was pointed out by one of my colleagues that the host wasn't activating because the rest of the networks were not configured, not the same as when we were running into the issues with nrpe.
I configured the networks and the host activated just fine.