Created attachment 1245853 [details] Engine and VDSM logs of this suite Description of problem: When running OST, sometimes hotplug_nic test fails because of the following exception: MainProcess|jsonrpc/0::DEBUG::2017-01-24 10:13:50,303::supervdsm_server::101::SuperVdsm.ServerCallback::(wrapper) return network_caps with {'bridges': {'ovirtmgmt': {'ipv6autoconf': True, 'addr': '192.168.201.2', 'dhcpv6': False, 'ipv6addrs': [], 'gateway': '192.168.201.1', 'dhcpv4': True, 'netmask': '255.255.255.0', 'ipv4defaultroute': True, 'stp': 'off', 'ipv4addrs': ['192.168.201.2/24'], 'mtu': '1500', 'ipv6gateway': '::', 'ports': ['eth0'], 'opts': {'multicast_last_member_count': '2', 'hash_elasticity': '4', 'multicast_query_response_interval': '1000', 'group_fwd_mask': '0x0', 'multicast_snooping': '1', 'multicast_startup_query_interval': '3125', 'hello_timer': '15', 'multicast_querier_interval': '25500', 'max_age': '2000', 'hash_max': '512', 'stp_state': '0', 'topology_change_detected': '0', 'priority': '32768', 'multicast_membership_interval': '26000', 'root_path_cost': '0', 'root_port': '0', 'multicast_querier': '0', 'multicast_startup_query_count': '2', 'nf_call_iptables': '0', 'topology_change': '0', 'hello_time': '200', 'root_id': '8000.5452c0a8c902', 'bridge_id': '8000.5452c0a8c902', 'topology_change_timer': '0', 'ageing_time': '30000', 'nf_call_ip6tables': '0', 'gc_timer': '20299', 'nf_call_arptables': '0', 'group_addr': '1:80:c2:0:0:0', 'multicast_last_member_interval': '100', 'default_pvid': '1', 'multicast_query_interval': '12500', 'tcn_timer': '0', 'multicast_router': '1', 'vlan_filtering': '0', 'forward_delay': '0'}}}, 'bondings': {}, 'nameservers': ['192.168.201.1'], 'nics': {'eth3': {'ipv6autoconf': True, 'addr': '192.168.202.212', 'ipv6gateway': 'fe80::5054:ff:fe65:57d6', 'dhcpv6': False, 'ipv6addrs': [], 'mtu': '1500', 'dhcpv4': True, 'netmask': '255.255.255.0', 'ipv4defaultroute': False, 'ipv4addrs': ['192.168.202.212/24'], 'hwaddr': '54:52:c0:a8:ca:03', 'speed': 0, 'gateway': '192.168.202.1'}, 'eth2': {'ipv6autoconf': True, 'addr': '192.168.202.211', 'ipv6gateway': '::', 'dhcpv6': False, 'ipv6addrs': [], 'mtu': '1500', 'dhcpv4': True, 'netmask': '255.255.255.0', 'ipv4defaultroute': False, 'ipv4addrs': ['192.168.202.211/24'], 'hwaddr': '54:52:c0:a8:ca:02', 'speed': 0, 'gateway': '192.168.202.1'}, 'eth1': {'ipv6autoconf': True, 'addr': '192.168.200.143', 'ipv6gateway': '::', 'dhcpv6': False, 'ipv6addrs': [], 'mtu': '1500', 'dhcpv4': True, 'netmask': '255.255.255.0', 'ipv4defaultroute': False, 'ipv4addrs': ['192.168.200.143/24'], 'hwaddr': '54:52:c0:a8:c8:02', 'speed': 0, 'gateway': '192.168.200.1'}, 'eth0': {'ipv6autoconf': False, 'addr': '', 'ipv6gateway': '::', 'dhcpv6': False, 'ipv6addrs': [], 'mtu': '1500', 'dhcpv4': False, 'netmask': '', 'ipv4defaultroute': False, 'ipv4addrs': [], 'hwaddr': '54:52:c0:a8:c9:02', 'speed': 0, 'gateway': ''}}, 'supportsIPv6': True, 'vlans': {}, 'networks': {'ovirtmgmt': {'dhcpv6': False, 'iface': 'ovirtmgmt', 'ipv6autoconf': True, 'addr': '192.168.201.2', 'bridged': True, 'ipv6addrs': [], 'switch': 'legacy', 'gateway': '192.168.201.1', 'dhcpv4': True, 'netmask': '255.255.255.0', 'ipv4defaultroute': True, 'stp': 'off', 'ipv4addrs': ['192.168.201.2/24'], 'mtu': '1500', 'ipv6gateway': '::', 'ports': ['eth0']}}} ifup/VLAN100_Network::DEBUG::2017-01-24 10:13:53,048::commands::93::root::(execCmd) FAILED: <err> = 'Running scope as unit 0e9c5843-c89e-499b-9057-1d70ea366504.scope.\n/etc/sysconfig/network-scripts/ifup-eth: line 297: 16535 Terminated /sbin/dhclient ${DHCLIENTARGS} ${DEVICE}\nCannot find device "VLAN100_Network"\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not exist.\n'; <rc> = 1 ifup/VLAN100_Network::ERROR::2017-01-24 10:13:53,049::concurrent::189::root::(run) FINISH thread <Thread(ifup/VLAN100_Network, started daemon 140683650258688)> failed Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/concurrent.py", line 185, in run ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/network/configurators/ifcfg.py", line 910, in _exec_ifup _exec_ifup_by_name(iface.name, cgroup) File "/usr/lib/python2.7/site-packages/vdsm/network/configurators/ifcfg.py", line 896, in _exec_ifup_by_name raise ConfigNetworkError(ERR_FAILED_IFUP, out[-1] if out else '') ConfigNetworkError: (29, 'Determining IPv6 information for VLAN100_Network... failed.') Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
Can you better define "sometimes"? How frequently does this occur?
(In reply to Leon Goldberg from comment #1) > Can you better define "sometimes"? How frequently does this occur? You need to use the NEEDINFO to get answers. For example, setting on you.
That was long time ago but we saw it happening this week as well. Please see dbelenky's emails.
The VDSM log errors seem not related to the OST test failure. If the OST failure the same as this one: http://jenkins.ovirt.org/view/experimental%20jobs/job/test-repo_ovirt_experimental_4.1/917/ Then this looks related to the jsonrpc connectivity between Engine and VDSM. The VDSM error logs seen here are related to issuing an IPv6 DHCP setup request which lago currently cannot serve. The flow goes something like this: 1. Setup request with DHCPv6 is sent to VDSM. 2. Setup succeeds but in a non-blocking mode (an ifup is run in the background). 3. After 13 sec, a new setup is issued which removes the network. 4. The network is removed with all its virtual devices, which in turn causes the still running ifup (from 2) to fail as the device it was running on was removed.
What's missing in Lago's DHCPv6? (it uses dnsmasq, perhaps mis-configured)?
(In reply to Yaniv Kaul from comment #5) > What's missing in Lago's DHCPv6? (it uses dnsmasq, perhaps mis-configured)? It's dnsmasq and VLANs, tracked by https://github.com/lago-project/lago/issues/375 There is no Vdsm bug here; the log noise is created by a broken test modify_host_ip_to_dhcp, which requests a dhcp address and does not wait for it to be acquired.