Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1479829

Summary: Can't set DHCP bootproto for non-VM network that is attached to a bond interface
Product: [oVirt] vdsm Reporter: Nikolai Sednev <nsednev>
Component: CoreAssignee: Edward Haas <edwardh>
Status: CLOSED CURRENTRELEASE QA Contact: Michael Burman <mburman>
Severity: high Docs Contact:
Priority: high    
Version: 4.19.27CC: bugs, danken, edwardh, mburman, nsednev
Target Milestone: ovirt-4.1.5Keywords: Regression
Target Release: ---Flags: rule-engine: ovirt-4.1+
rule-engine: blocker+
nsednev: testing_plan_complete?
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-08-23 08:04:22 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Network RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
screencast
none
sosreport from the engine
none
supervdsm log none

Description Nikolai Sednev 2017-08-09 13:28:37 UTC
Description of problem:
802.1Q, none-required and none-vm network not getting IP address if being attached to bond0 interface.

The same network will receive IP address from DHCP if being set as vm-network.

The none-vm networks being used for MPIO storage tasks, aka iSCSI bond.

Version-Release number of selected component (if applicable):
On host:
libvirt-client-3.2.0-14.el7_4.2.x86_64
ovirt-vmconsole-1.0.4-1.el7ev.noarch
ovirt-hosted-engine-ha-2.1.4-1.el7ev.noarch
sanlock-3.5.0-1.el7.x86_64
ovirt-setup-lib-1.1.3-1.el7ev.noarch
ovirt-engine-sdk-python-3.6.9.1-1.el7ev.noarch
ovirt-vmconsole-host-1.0.4-1.el7ev.noarch
mom-0.5.9-1.el7ev.noarch
ovirt-imageio-daemon-1.0.0-0.el7ev.noarch
ovirt-hosted-engine-setup-2.1.3.5-1.el7ev.noarch
ovirt-host-deploy-1.6.6-1.el7ev.noarch
qemu-kvm-rhev-2.9.0-16.el7_4.3.x86_64
ovirt-imageio-common-1.0.0-0.el7ev.noarch
vdsm-4.19.26-1.el7ev.x86_64
Linux version 3.10.0-693.el7.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-16) (GCC) ) #1 SMP Thu Jul 6 19:56:57 EDT 2017
Linux 3.10.0-693.el7.x86_64 #1 SMP Thu Jul 6 19:56:57 EDT 2017 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux Server release 7.4 (Maipo)

On engine:
rhev-guest-tools-iso-4.1-5.el7ev.noarch
rhevm-dependencies-4.1.1-1.el7ev.noarch
rhevm-doc-4.1.3-1.el7ev.noarch
rhevm-branding-rhev-4.1.0-2.el7ev.noarch
rhevm-4.1.3.5-0.1.el7.noarch
rhevm-setup-plugins-4.1.2-1.el7ev.noarch
Linux version 3.10.0-514.21.2.el7.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) ) #1 SMP Sun May 28 17:08:21 EDT 2017
Linux 3.10.0-514.21.2.el7.x86_64 #1 SMP Sun May 28 17:08:21 EDT 2017 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux Server release 7.3 (Maipo)

How reproducible:
100%

Steps to Reproduce:
1.In my scenario I've used SHE, but this is happening also on regular environment, you should have bonded interface with 802.1Q tagging configured on it.
2.Create network as none-required network and none-vm network.
3.Attach network from UI to bonded and tagged interface and set it to DHCP.

Actual results:
"Error while executing action Setup Networks: unexpected exception."
Looks like its VDSM bug.

Expected results:
Attached network to tagged interface should receive IP address from DHCP server for none-vm network just like its done for vm-network.

Additional info:
Logs and screencast being attached.

Comment 1 Nikolai Sednev 2017-08-09 13:32:47 UTC
Created attachment 1311233 [details]
screencast

Comment 2 Nikolai Sednev 2017-08-09 13:39:27 UTC
Created attachment 1311234 [details]
sosreport from the engine

Comment 3 Nikolai Sednev 2017-08-09 13:42:03 UTC
Sosreport from host alma04 available from here https://drive.google.com/a/redhat.com/file/d/0B85BEaDBcF88enRrTmQyMFZwZFE/view?usp=sharing

Comment 4 Michael Burman 2017-08-09 13:47:24 UTC
Please note that the only scenario in which this is reproduced is when trying to set DHCP boot protocol for a non-VM network that is attached to a bond interface, failing with on vdsm side:

2017-08-09 16:45:07,828+0300 ERROR (jsonrpc/6) [jsonrpc.JsonRpcServer] Internal server error (__init__:642)
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 637, in _handle_request
    res = method(**params)
  File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 201, in _dynamicMethod
    result = fn(*methodArgs)
  File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1409, in setupNetworks
    supervdsm.getProxy().setupNetworks(networks, bondings, options)
  File "/usr/lib/python2.7/site-packages/vdsm/supervdsm.py", line 53, in __call__
    return callMethod()
  File "/usr/lib/python2.7/site-packages/vdsm/supervdsm.py", line 51, in <lambda>
    **kwargs)
  File "<string>", line 2, in setupNetworks
  File "/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in _callmethod
    raise convert_to_error(kind, result)
IOError: [Errno 19] bond0.162 is not present in the system
2017-08-09 16:45:07,829+0300 INFO  (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call Host.setupNetworks failed (error -32603) in 15.51 seconds (__init__:604)

Comment 5 Dan Kenigsberg 2017-08-09 14:22:39 UTC
Nikolai, can you attach supervdsm.log?

Michael, do you see this as well, or is it specific to Nikolai's host?

Comment 6 Michael Burman 2017-08-09 14:25:58 UTC
(In reply to Dan Kenigsberg from comment #5)
> Nikolai, can you attach supervdsm.log?
> 
> Michael, do you see this as well, or is it specific to Nikolai's host?

It's not Nikolai's spesific host, it is easily reproduced with next steps:

1) Create bond via setup networks
2) Attach non-VM network to the bond
3) Try to set dhcp bootproto for the non-vm network

Result - failed with error on vdsm side:
IOError: [Errno 19] bond0.162 is not present in the system

I will attach the log

Comment 7 Michael Burman 2017-08-09 14:29:35 UTC
Created attachment 1311240 [details]
supervdsm log

Comment 8 Nikolai Sednev 2017-08-09 18:27:44 UTC
(In reply to Dan Kenigsberg from comment #5)
> Nikolai, can you attach supervdsm.log?
> 
> Michael, do you see this as well, or is it specific to Nikolai's host?

Everything was already attached in https://bugzilla.redhat.com/show_bug.cgi?id=1479829#c3, also there is a screencast attached to this bug which exactly shows what and how to reproduce.

Comment 9 Dan Kenigsberg 2017-08-10 12:31:08 UTC
(In reply to Nikolai Sednev from comment #8)
> (In reply to Dan Kenigsberg from comment #5)
> > Nikolai, can you attach supervdsm.log?
> > 
> > Michael, do you see this as well, or is it specific to Nikolai's host?
> 
> Everything was already attached in
> https://bugzilla.redhat.com/show_bug.cgi?id=1479829#c3, also there is a
> screencast attached to this bug which exactly shows what and how to
> reproduce.

As specically-requested log attachement is more helpful than an off-bz dump of everything. But Burman has already done that.

Comment 10 Yaniv Kaul 2017-08-13 07:47:03 UTC
How could it be a blocker but medium severity?

Comment 11 Michael Burman 2017-08-16 05:38:59 UTC
Verified on - vdsm-4.19.28-1.el7ev.x86_64

Comment 12 Nikolai Sednev 2017-08-16 12:43:12 UTC
I still see this issue happening on ovirt-engine-4.2.0-0.0.master.20170811144920.gita423008.el7.centos.noarch on hosts with vdsm-4.20.2-74.git2be4775.el7.centos.x86_64.
Should this fix get cloned to the master any soon?

Comment 13 Michael Burman 2017-08-16 12:51:33 UTC
(In reply to Nikolai Sednev from comment #12)
> I still see this issue happening on
> ovirt-engine-4.2.0-0.0.master.20170811144920.gita423008.el7.centos.noarch on
> hosts with vdsm-4.20.2-74.git2be4775.el7.centos.x86_64.
> Should this fix get cloned to the master any soon?

Nikolai, 
This was fixed for master and working as expected on vdsm-4.20.2-74.git2be4775.el7.centos.x86_64

Comment 14 Nikolai Sednev 2017-08-16 13:03:16 UTC
(In reply to Michael Burman from comment #13)
> (In reply to Nikolai Sednev from comment #12)
> > I still see this issue happening on
> > ovirt-engine-4.2.0-0.0.master.20170811144920.gita423008.el7.centos.noarch on
> > hosts with vdsm-4.20.2-74.git2be4775.el7.centos.x86_64.
> > Should this fix get cloned to the master any soon?
> 
> Nikolai, 
> This was fixed for master and working as expected on
> vdsm-4.20.2-74.git2be4775.el7.centos.x86_64

Working now, after clicking on "Refresh Capabilities" on host.