Bug 1825740

Summary:	[OVN] ovirt sync to ovirt-provider-ovn breaks on upgrade of ovirt-engine-4.2.8.9-0.1.el7ev.noarch to ovirt-engine-4.3.9.4-11.el7.noarch
Product:	[oVirt] ovirt-engine	Reporter:	msheena
Component:	Setup.Engine.OVN	Assignee:	Dominik Holler <dholler>
Status:	CLOSED CURRENTRELEASE	QA Contact:	msheena
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	4.3.9.4	CC:	aoconnor, bugs, danken, dholler, dkenigsb, mavital, mburman, royoung
Target Milestone:	ovirt-4.3.10	Keywords:	Automation, Regression
Target Release:	4.3.10	Flags:	pm-rhel: ovirt-4.3+ aoconnor: blocker+
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:	ovirt-engine-4.3.10	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2020-06-10 10:57:03 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	Network	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description msheena 2020-04-20 05:57:59 UTC

Description of problem
======================
On upgrade of oVirt from ovirt-engine-4.2.8.9-0.1.el7ev.noarch to ovirt-engine-4.3.9.4-11.el7.noarch, oVirt loses sync to ovirt-provider-ovn.

From /var/log/ovirt-engine/engine.log
-------------------------------------
2020-04-20 08:20:27,548+03 ERROR [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-24) [5e2bcebc] Command 'org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand' failed: EngineException: (Failed with error Internal Server Error and code 5050)


Version-Release number of selected component (if applicable)
============================================================
ovirt-engine-4.2.8.9-0.1.el7ev.noarch
ovirt-engine-4.3.9.4-11.el7.noarch
ovirt-provider-ovn-1.2.29-1.el7ev.noarch

How reproducible
================
Reproduced in 4.2 -> 4.3 upgrade flow (skipping hosts upgrade)

Steps to Reproduce
==================
1. Deploy ovirt-engine-4.2.8.9-0.1.el7ev.noarch and install hosts with rhvh-4.2.20.1-0.20200416.0+1
2. Upgrade oVirt to ovirt-engine-4.3.9.4-11.el7.noarch (without upgrading hosts)

Actual results
==============
oVirt cannot sync with ovirt-provider-ovn

Expected results
================
oVirt syncs with ovirt-provider-ovn

Additional info
===============

After upgrade
-------------
ovirt-provider-ovn.service is running on engine VM
ovn-controller.service is running on hosts, but displays the following:

Apr 19 19:33:24 lynx18.lab.eng.tlv2.redhat.com ovn-controller[58261]: ovs|00009|reconnect|ERR|ssl:<engine VM IP>:6642: no response to inactivity probe after 5 seconds, disconnecting
Apr 19 19:33:25 lynx18.lab.eng.tlv2.redhat.com ovn-controller[58261]: ovs|00011|stream_ssl|ERR|ssl:<engine VM IP>:6642: connect: Network is unreachable
Apr 19 19:33:27 lynx18.lab.eng.tlv2.redhat.com ovn-controller[58261]: ovs|00015|stream_ssl|ERR|ssl:<engine VM IP>:6642: connect: Network is unreachable
Apr 19 19:33:31 lynx18.lab.eng.tlv2.redhat.com ovn-controller[58261]: ovs|00019|stream_ssl|ERR|ssl:<engine VM IP>:6642: connect: Network is unreachable

Comment 4 Dominik Holler 2020-04-20 09:03:02 UTC

Even engine-setup sets the correct file permissions for OVN FDP 20.A, engine-setup in ovirt-4.3 does not restart ovn-northd on upgrade,
which seems to be required to let OVN detect the new file permissions.
For this reasons this bug is a problem in rhv-4.3, but not in ovirt-4.2.
The workaround is to restart OVN on RHVM machine manually, after the execution of engine-setup:
systemctl restart ovn-northd.service

I will check backporting
https://gerrit.ovirt.org/#/c/107847/
to ovirt-4.3 would fix the problem.

Comment 5 RHEL Program Management 2020-04-21 09:15:57 UTC

This bug report has Keywords: Regression or TestBlocker.
Since no regressions or test blockers are allowed between releases, it is also being identified as a blocker for this release. Please resolve ASAP.

Comment 6 msheena 2020-04-30 14:03:21 UTC

On upgrading RHV 4.2.8.9 to 4.3.10 the manager was still unable to communicate with ovirt-provider-ovn

Comment 7 RHEL Program Management 2020-04-30 14:03:27 UTC

Target release should be placed once a package build is known to fix a issue. Since this bug is not modified, the target version has been reset. Please use target milestone to plan a fix for a oVirt release.

Comment 8 msheena 2020-05-17 07:36:50 UTC

Verified on
===========
ovirt-engine-4.3.10.3-0.1.master.el7.noarch

Comment 9 Michal Skrivanek 2020-06-10 10:57:03 UTC

4.3.10 was released