Bug 1417595 - hotplug_nic test fails in OST
Summary: hotplug_nic test fails in OST
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: vdsm
Classification: oVirt
Component: SuperVDSM
Version: 4.20.0
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ovirt-4.2.0
: ---
Assignee: Leon Goldberg
QA Contact: Meni Yakove
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-01-30 11:37 UTC by gshinar
Modified: 2017-04-12 07:51 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-03-12 15:12:54 UTC
oVirt Team: Network
edwardh: needinfo-
rule-engine: ovirt-4.2+


Attachments (Terms of Use)
Engine and VDSM logs of this suite (1.79 MB, application/x-gzip)
2017-01-30 11:37 UTC, gshinar
no flags Details

Description gshinar 2017-01-30 11:37:22 UTC
Created attachment 1245853 [details]
Engine and VDSM logs of this suite

Description of problem:
When running OST, sometimes hotplug_nic test fails because of the following exception:
MainProcess|jsonrpc/0::DEBUG::2017-01-24 10:13:50,303::supervdsm_server::101::SuperVdsm.ServerCallback::(wrapper) return network_caps with {'bridges': {'ovirtmgmt': {'ipv6autoconf': True, 'addr': '192.168.201.2', 'dhcpv6': False, 'ipv6addrs': [], 'gateway': '192.168.201.1', 'dhcpv4': True, 'netmask': '255.255.255.0', 'ipv4defaultroute': True, 'stp': 'off', 'ipv4addrs': ['192.168.201.2/24'], 'mtu': '1500', 'ipv6gateway': '::', 'ports': ['eth0'], 'opts': {'multicast_last_member_count': '2', 'hash_elasticity': '4', 'multicast_query_response_interval': '1000', 'group_fwd_mask': '0x0', 'multicast_snooping': '1', 'multicast_startup_query_interval': '3125', 'hello_timer': '15', 'multicast_querier_interval': '25500', 'max_age': '2000', 'hash_max': '512', 'stp_state': '0', 'topology_change_detected': '0', 'priority': '32768', 'multicast_membership_interval': '26000', 'root_path_cost': '0', 'root_port': '0', 'multicast_querier': '0', 'multicast_startup_query_count': '2', 'nf_call_iptables': '0', 'topology_change': '0', 'hello_time': '200', 'root_id': '8000.5452c0a8c902', 'bridge_id': '8000.5452c0a8c902', 'topology_change_timer': '0', 'ageing_time': '30000', 'nf_call_ip6tables': '0', 'gc_timer': '20299', 'nf_call_arptables': '0', 'group_addr': '1:80:c2:0:0:0', 'multicast_last_member_interval': '100', 'default_pvid': '1', 'multicast_query_interval': '12500', 'tcn_timer': '0', 'multicast_router': '1', 'vlan_filtering': '0', 'forward_delay': '0'}}}, 'bondings': {}, 'nameservers': ['192.168.201.1'], 'nics': {'eth3': {'ipv6autoconf': True, 'addr': '192.168.202.212', 'ipv6gateway': 'fe80::5054:ff:fe65:57d6', 'dhcpv6': False, 'ipv6addrs': [], 'mtu': '1500', 'dhcpv4': True, 'netmask': '255.255.255.0', 'ipv4defaultroute': False, 'ipv4addrs': ['192.168.202.212/24'], 'hwaddr': '54:52:c0:a8:ca:03', 'speed': 0, 'gateway': '192.168.202.1'}, 'eth2': {'ipv6autoconf': True, 'addr': '192.168.202.211', 'ipv6gateway': '::', 'dhcpv6': False, 'ipv6addrs': [], 'mtu': '1500', 'dhcpv4': True, 'netmask': '255.255.255.0', 'ipv4defaultroute': False, 'ipv4addrs': ['192.168.202.211/24'], 'hwaddr': '54:52:c0:a8:ca:02', 'speed': 0, 'gateway': '192.168.202.1'}, 'eth1': {'ipv6autoconf': True, 'addr': '192.168.200.143', 'ipv6gateway': '::', 'dhcpv6': False, 'ipv6addrs': [], 'mtu': '1500', 'dhcpv4': True, 'netmask': '255.255.255.0', 'ipv4defaultroute': False, 'ipv4addrs': ['192.168.200.143/24'], 'hwaddr': '54:52:c0:a8:c8:02', 'speed': 0, 'gateway': '192.168.200.1'}, 'eth0': {'ipv6autoconf': False, 'addr': '', 'ipv6gateway': '::', 'dhcpv6': False, 'ipv6addrs': [], 'mtu': '1500', 'dhcpv4': False, 'netmask': '', 'ipv4defaultroute': False, 'ipv4addrs': [], 'hwaddr': '54:52:c0:a8:c9:02', 'speed': 0, 'gateway': ''}}, 'supportsIPv6': True, 'vlans': {}, 'networks': {'ovirtmgmt': {'dhcpv6': False, 'iface': 'ovirtmgmt', 'ipv6autoconf': True, 'addr': '192.168.201.2', 'bridged': True, 'ipv6addrs': [], 'switch': 'legacy', 'gateway': '192.168.201.1', 'dhcpv4': True, 'netmask': '255.255.255.0', 'ipv4defaultroute': True, 'stp': 'off', 'ipv4addrs': ['192.168.201.2/24'], 'mtu': '1500', 'ipv6gateway': '::', 'ports': ['eth0']}}}
ifup/VLAN100_Network::DEBUG::2017-01-24 10:13:53,048::commands::93::root::(execCmd) FAILED: <err> = 'Running scope as unit 0e9c5843-c89e-499b-9057-1d70ea366504.scope.\n/etc/sysconfig/network-scripts/ifup-eth: line 297: 16535 Terminated              /sbin/dhclient ${DHCLIENTARGS} ${DEVICE}\nCannot find device "VLAN100_Network"\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\n'; <rc> = 1
ifup/VLAN100_Network::ERROR::2017-01-24 10:13:53,049::concurrent::189::root::(run) FINISH thread <Thread(ifup/VLAN100_Network, started daemon 140683650258688)> failed
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/vdsm/concurrent.py", line 185, in run
    ret = func(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/vdsm/network/configurators/ifcfg.py", line 910, in _exec_ifup
    _exec_ifup_by_name(iface.name, cgroup)
  File "/usr/lib/python2.7/site-packages/vdsm/network/configurators/ifcfg.py", line 896, in _exec_ifup_by_name
    raise ConfigNetworkError(ERR_FAILED_IFUP, out[-1] if out else '')
ConfigNetworkError: (29, 'Determining IPv6 information for VLAN100_Network... failed.')


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 1 Leon Goldberg 2017-02-02 13:30:15 UTC
Can you better define "sometimes"? How frequently does this occur?

Comment 2 Yaniv Kaul 2017-03-09 08:02:59 UTC
(In reply to Leon Goldberg from comment #1)
> Can you better define "sometimes"? How frequently does this occur?

You need to use the NEEDINFO to get answers. For example, setting on you.

Comment 3 gshinar 2017-03-09 08:12:05 UTC
That was long time ago but we saw it happening this week as well. 
Please see dbelenky's emails.

Comment 4 Edward Haas 2017-03-09 14:12:51 UTC
The VDSM log errors seem not related to the OST test failure.

If the OST failure the same as this one: http://jenkins.ovirt.org/view/experimental%20jobs/job/test-repo_ovirt_experimental_4.1/917/
Then this looks related to the jsonrpc connectivity between Engine and VDSM.

The VDSM error logs seen here are related to issuing an IPv6 DHCP setup request which lago currently cannot serve.
The flow goes something like this:
1. Setup request with DHCPv6 is sent to VDSM.
2. Setup succeeds but in a non-blocking mode (an ifup is run in the background).
3. After 13 sec, a new setup is issued which removes the network.
4. The network is removed with all its virtual devices, which in turn causes the still running ifup (from 2) to fail as the device it was running on was removed.

Comment 5 Yaniv Kaul 2017-03-09 14:19:38 UTC
What's missing in Lago's DHCPv6? (it uses dnsmasq, perhaps mis-configured)?

Comment 6 Dan Kenigsberg 2017-03-12 15:12:54 UTC
(In reply to Yaniv Kaul from comment #5)
> What's missing in Lago's DHCPv6? (it uses dnsmasq, perhaps mis-configured)?

It's dnsmasq and VLANs, tracked by https://github.com/lago-project/lago/issues/375

There is no Vdsm bug here; the log noise is created by a broken test modify_host_ip_to_dhcp, which requests a dhcp address and does not wait for it to be acquired.


Note You need to log in before you can comment on or make changes to this bug.