Bug 1292364 - Engine reports Network error after changing the management network bootproto from dhcp to static. although the change took place
Engine reports Network error after changing the management network bootproto ...
Status: CLOSED WONTFIX
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-engine (Show other bugs)
3.5.7
x86_64 Linux
unspecified Severity medium
: ovirt-3.5.8
: ---
Assigned To: Alona Kaplan
Michael Burman
network
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-12-17 03:27 EST by Michael Burman
Modified: 2016-01-26 03:07 EST (History)
10 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-01-25 08:35:58 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: Network
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
engine log (437.37 KB, application/x-gzip)
2015-12-17 03:27 EST, Michael Burman
no flags Details
vdsm logs (447.42 KB, application/x-gzip)
2015-12-17 06:01 EST, Michael Burman
no flags Details

  None (edit)
Description Michael Burman 2015-12-17 03:27:25 EST
Created attachment 1106601 [details]
engine log

Description of problem:
Changing the management network bootproto from dhcp to static ip and vise versa, ends up with "Error while executing action Setup Networks: Network error during communication with the Host." although the change took place in UI - setup networks dialog, in ifcfg-rhevm file and in vdsCaps:

1) Change the rhevm's bootproto from dhcp to static ip:
- error is thrown after few seconds
- UI report static ip
- [root@pink-vds2 ~]# cat /etc/sysconfig/network-scripts/ifcfg-rhevm 
# Generated by VDSM version 4.16.31-1.el7ev
IPADDR=10.35.128.19
NETMASK=255.255.255.0
GATEWAY=10.35.128.254
BOOTPROTO=none
- vdscaps report:
'rhevm': {'addr': '10.35.128.19',
                              'bootproto4': 'none',
                              'bridged': True,
                              'cfg': {'BOOTPROTO': 'none',

2) Change the rhevm's bootproto from static ip back to dhcp:
- error is thrown after few seconds
- UI reports dhcp
- [root@pink-vds2 ~]# cat /etc/sysconfig/network-scripts/ifcfg-rhevm 
# Generated by VDSM version 4.16.31-1.el7ev
BOOTPROTO=dhcp
- vdsCaps report:
'rhevm': {'addr': '10.35.128.19',
                              'bootproto4': 'dhcp',
                              'bridged': True,
                              'cfg': {'BOOTPROTO': 'dhcp',

engine.log:

2015-12-17 10:02:17,243 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (ajp-/127.0.0.1:8702-6) [3eb7e877] org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNe
tworkException: Heartbeat exeeded
2015-12-17 10:02:17,262 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-68) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkExce
ption: VDSGenericException: VDSNetworkException: Heartbeat exeeded


2015-12-17 10:02:19,679 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-64) [4ff7c13b] ERROR, org.ovirt.engine.core.vdsbroker.irsbroker.GetStoragePoolInfoVDSCommand, exception: XmlRpcRunTimeException: Connection issues during send request, log id: 2fbbd562: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: Connection issues during send request


2015-12-17 10:02:27,425 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-22) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout


Version-Release number of selected component (if applicable):
3.5.7-0.1.el6ev
vdsm 4.16.31-1.el7ev

How reproducible:
100

Steps to Reproduce:
1. Change rhevm's bootproto from dhcp to static ip and vise versa

Actual results:
- Operation canceled
"Error while executing action Setup Networks: Network error during communication with the Host."
Change took place in:
UI - setup networks dialog
ifcfg-rhevm
vdsCaps

Expected results:
No error message should be displayed.
It's a regression from previous version 3.5.6
 

Additional info:
Maybe relevant to the fix in BZ 1279824
Comment 1 Dan Kenigsberg 2015-12-17 04:47:01 EST
What Engine version is it? Does the bug show up when you are editing non-management network?
Comment 2 Michael Burman 2015-12-17 04:50:47 EST
Hi Dan

engine version is provided in description and bug verson ^^ 3.5.7-0.1.el6ev

No, only for management network.
Comment 3 Dan Kenigsberg 2015-12-17 05:10:23 EST
Can you attach super,vdsm,connectivity.log as well?
Did Vdsm persist the new config?
Comment 4 Michael Burman 2015-12-17 06:01 EST
Created attachment 1106693 [details]
vdsm logs
Comment 5 Michael Burman 2015-12-17 06:01:23 EST
It seems that vdsm is not persist the new config -->

'rhevm': {'addr': '10.35.128.19',
                              'bootproto4': 'none',
                              'bridged': True,
                              'cfg': {'BOOTPROTO': 'none'

[root@pink-vds2 ~]# cat /var/lib/vdsm/persistence/netconf/nets/rhevm 
{"nic": "enp4s0", "mtu": "1500", "bootproto": "dhcp", "STP": "no", "bridged": "true", "defaultRoute": true}

[root@pink-vds2 ~]# cat /var/run/vdsm/netconf/nets/rhevm 
{"nic": "enp4s0", "ipaddr": "10.35.128.19", "mtu": "1500", "netmask": "255.255.255.0", "STP": "no", "bridged": "true", "gateway": "10.35.128.254", "defaultRoute": true}

Attaching logs.
Comment 6 Michael Burman 2015-12-17 06:07:03 EST
Important thing i didn't mentioned, the operation cause the server to become non responsive for few seconds, host and engine loosing connectivity for few seconds.
Comment 8 Dan Kenigsberg 2015-12-21 08:35:56 EST
does the bug show up with engine-3.5.6 and new Vdsm?
An engine-3.5.7 and and old Vdsm?

I don't have a clue were the regression may lie (Engine/Vdsm)

lowering severity as editing management network is not recommended anyway.
Comment 9 Michael Burman 2015-12-21 09:09:58 EST
Hi Dan, 

The bug doesn't show up in engine 3.5.6 and vdsm 3.5.7.
Managed to reproduce in engine 3.5.7 with older vdsm versions like :
vdsm  4.16.30-1.el7ev.x86_64 and 4.16.28-1.el7ev.x86_64 (reproduced not 100%)

Dan, please note, that this report no longer 100 reproducible, because i had several attempts which succeeded. But it is still failing in 60-70% of the attempts.
Comment 10 Dan Kenigsberg 2015-12-31 08:16:27 EST
Alona, is there anything new in Engine-3.5.7 that might have triggered this bug?
Comment 11 RHEL Product and Program Management 2016-01-25 08:35:58 EST
Development Management has reviewed and declined this request.
You may appeal this decision by reopening this request.
Comment 12 Dan Kenigsberg 2016-01-26 03:07:01 EST
Alona has verified that nothing has changed since ovirt-3.5.6 that could have triggered this bug. Moreover, as it reproduces very inconsistently, we believe it is not a regression. It seems like an issue that has been lurking since the introduction of jsonrpc, and should be fixed with bug 1277988 which would stop Engine's change of jsonrpc connection during setupNetworks command.

I do not believe this issue is urgent enough to merit entry to 3.5.8, hence my devel_ack-.

Note You need to log in before you can comment on or make changes to this bug.