Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1534258

Summary: Cannot add host to RHV with latest rhel7.5: dump_bonding_options races with NM
Product: [oVirt] vdsm Reporter: Michael Burman <mburman>
Component: CoreAssignee: Petr Horáček <phoracek>
Status: CLOSED CURRENTRELEASE QA Contact: Michael Burman <mburman>
Severity: high Docs Contact:
Priority: low    
Version: 4.20.10CC: bugs, danken, edwardh, khakimi, michal.skrivanek, myakove, ncredi, phoracek, pmatyas, ratamir, ylavi
Target Milestone: ovirt-4.2.2Keywords: AutomationBlocker, Regression
Target Release: ---Flags: rule-engine: ovirt-4.2+
rule-engine: blocker+
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-03-29 11:11:31 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Network RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1534483, 1538630    
Bug Blocks:    
Attachments:
Description Flags
no vdsm log as it wasn't started
none
ovirt host deploy none

Description Michael Burman 2018-01-14 15:27:33 UTC
Description of problem:
Can't add host to RHV with latest kernel rhel7.5 - 3.10.0-829.el7.x86_64

Trying to add host with latest 7.5 kernel is failing and vdsmd can't start. 
Possibly related to the latest changes done on the kernel.

Version-Release number of selected component (if applicable):
kernel-3.10.0-829.el7.x86_64

How reproducible:
Seems to be 100% 

Steps to Reproduce:
1. Install latest rhel7.5 with latest kernel kernel-3.10.0-829.el7.x86_64
Try to add host to intel-conroe, westmere type cluster

Actual results:
Install failed - vdsmd can't start

Expected results:
Should work

Comment 1 Michael Burman 2018-01-14 15:31:31 UTC
Created attachment 1381078 [details]
no vdsm log as it wasn't started

Comment 2 Michael Burman 2018-01-14 15:47:10 UTC
Created attachment 1381081 [details]
ovirt host deploy

Comment 3 Michal Skrivanek 2018-01-15 07:41:15 UTC
failing in vdsm network initialization

Comment 4 Michael Burman 2018-01-15 07:46:14 UTC
Seems to be related to the bondscan and the new NetworkManager 1:1.10.2-4.el7

Jan 14 17:12:41 lynx23.qa.lab.tlv.redhat.com vdsm-tool[31505]: Traceback (most recent call last):
Jan 14 17:12:41 lynx23.qa.lab.tlv.redhat.com vdsm-tool[31505]: File "/usr/bin/vdsm-tool", line 219, in main
Jan 14 17:12:41 lynx23.qa.lab.tlv.redhat.com vdsm-tool[31505]: return tool_command[cmd]["command"](*args)
Jan 14 17:12:41 lynx23.qa.lab.tlv.redhat.com vdsm-tool[31505]: File "/usr/lib/python2.7/site-packages/vdsm/tool/network.py", line 94, in dump_bonding_options
Jan 14 17:12:41 lynx23.qa.lab.tlv.redhat.com vdsm-tool[31505]: sysfs_options_mapper.dump_bonding_options()
Jan 14 17:12:41 lynx23.qa.lab.tlv.redhat.com vdsm-tool[31505]: File "/usr/lib/python2.7/site-packages/vdsm/network/link/bond/sysfs_options_mapper.py", line 47, in dump_bond
Jan 14 17:12:41 lynx23.qa.lab.tlv.redhat.com vdsm-tool[31505]: jdump(_get_default_bonding_options(), f)
Jan 14 17:12:41 lynx23.qa.lab.tlv.redhat.com vdsm-tool[31505]: File "/usr/lib/python2.7/site-packages/vdsm/network/link/bond/sysfs_options_mapper.py", line 69, in _get_defa
Jan 14 17:12:41 lynx23.qa.lab.tlv.redhat.com vdsm-tool[31505]: with _bond_device(bond_name, mode):
Jan 14 17:12:41 lynx23.qa.lab.tlv.redhat.com vdsm-tool[31505]: File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
Jan 14 17:12:41 lynx23.qa.lab.tlv.redhat.com vdsm-tool[31505]: return self.gen.next()
Jan 14 17:12:41 lynx23.qa.lab.tlv.redhat.com vdsm-tool[31505]: File "/usr/lib/python2.7/site-packages/vdsm/network/link/bond/sysfs_options_mapper.py", line 102, in _bond_de
Jan 14 17:12:41 lynx23.qa.lab.tlv.redhat.com vdsm-tool[31505]: _change_mode(bond_name, mode)
Jan 14 17:12:41 lynx23.qa.lab.tlv.redhat.com vdsm-tool[31505]: File "/usr/lib/python2.7/site-packages/vdsm/network/link/bond/sysfs_options_mapper.py", line 112, in _change_
Jan 14 17:12:41 lynx23.qa.lab.tlv.redhat.com vdsm-tool[31505]: opt.write(mode)
Jan 14 17:12:41 lynx23.qa.lab.tlv.redhat.com vdsm-tool[31505]: IOError: [Errno 22] Invalid argument

Comment 5 Meni Yakove 2018-01-15 08:47:56 UTC
yes, it looks like a race between vdsm's bondscan and NetworkManager.

Jan 15 09:33:44 lynx23.qa.lab.tlv.redhat.com kernel: bonding: bondscan-oY0pGe is being created...
Jan 15 09:33:44 lynx23.qa.lab.tlv.redhat.com vdsm-tool[46702]: Traceback (most recent call last):
Jan 15 09:33:44 lynx23.qa.lab.tlv.redhat.com vdsm-tool[46702]: File "/usr/bin/vdsm-tool", line 219, in main
Jan 15 09:33:44 lynx23.qa.lab.tlv.redhat.com NetworkManager[3482]: <info>  [1516001624.8759] manager: (bondscan-oY0pGe): new Bond device (/org/freedesktop/NetworkManager/De
Jan 15 09:33:44 lynx23.qa.lab.tlv.redhat.com vdsm-tool[46702]: return tool_command[cmd]["command"](*args)

Comment 6 Michael Burman 2018-01-22 06:24:24 UTC
On host reboot vdsm won't start as well because of this bug.

Comment 7 Petr Horáček 2018-01-25 10:13:59 UTC
It failed on upgrade tests.

Comment 8 Meni Yakove 2018-01-29 08:54:09 UTC
*** Bug 1537673 has been marked as a duplicate of this bug. ***

Comment 9 Edward Haas 2018-01-30 14:15:07 UTC
The suggested workaround is to unmanage a bond device immediately after creating it.
We have been hold back by an incident where the device cannot be unmanaged, raising an access deny error.

As the workaround is not that nice, I suggest waiting for NM team to respond on https://bugzilla.redhat.com/show_bug.cgi?id=1538630

Comment 10 Red Hat Bugzilla Rules Engine 2018-02-14 08:48:40 UTC
This bug report has Keywords: Regression or TestBlocker.
Since no regressions or test blockers are allowed between releases, it is also being identified as a blocker for this release. Please resolve ASAP.

Comment 11 Dan Kenigsberg 2018-02-21 08:50:33 UTC
Hopefully fixed by kernel.

Comment 12 Michael Burman 2018-02-27 09:02:12 UTC
Verified on - vdsm-4.20.19-1.el7ev.x86_64 and 4.2.2.1-0.1.el7 
kerne-3.10.0-855.el7.x86_64

Comment 13 Sandro Bonazzola 2018-03-29 11:11:31 UTC
This bugzilla is included in oVirt 4.2.2 release, published on March 28th 2018.

Since the problem described in this bug report should be
resolved in oVirt 4.2.2 release, it has been closed with a resolution of CURRENT RELEASE.

If the solution does not work for you, please open a new bug report.