1625355 – updating to openvswitch 2.10 results in openvswitch service getting shut down

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1625355 - updating to openvswitch 2.10 results in openvswitch service getting shut down

Summary: updating to openvswitch 2.10 results in openvswitch service getting shut down

Keywords:
Status:	CLOSED NOTABUG
Alias:	None
Product:	Red Hat Enterprise Linux 7
Classification:	Red Hat
Component:	openvswitch2.10
Sub Component:
Version:	7.6
Hardware:	Unspecified
OS:	Unspecified
Priority:	high
Severity:	high
Target Milestone:	rc
Target Release:	---
Assignee:	Flavio Leitner
QA Contact:	Rick Alongi
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2018-09-04 17:26 UTC by Rick Alongi
Modified:	2018-09-18 17:23 UTC (History)
CC List:	7 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2018-09-12 00:53:31 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Bugzilla	1628908	0	high	CLOSED	[UPGRADES][14] OSP 13 to 14 upgrade fails due to openvswitch package rename in ControllerAllNodesValidationDeployment' s...	2021-02-22 00:41:40 UTC

Internal Links: 1628908

Description Rick Alongi 2018-09-04 17:26:10 UTC

Description of problem:

updating to openvswitch 2.10 results in openvswitch service getting shut down

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1. 
2.
3.

Actual results:


Expected results:


Additional info:

[root@netqe12 tools]# rpm -qa | grep openvswitch
openvswitch-2.9.0-56.el7fdp.x86_64
kernel-kernel-networking-openvswitch-upgrade_ovs-1.1-5.noarch
openvswitch-selinux-extra-policy-1.0-3.el7fdp.noarch

[root@netqe12 tools]# systemctl status openvswitch
● openvswitch.service - Open vSwitch
   Loaded: loaded (/usr/lib/systemd/system/openvswitch.service; disabled; vendor preset: disabled)
   Active: active (exited) since Tue 2018-09-04 13:15:35 EDT; 1min 7s ago
  Process: 60203 ExecStart=/bin/true (code=exited, status=0/SUCCESS)
 Main PID: 60203 (code=exited, status=0/SUCCESS)

Sep 04 13:15:35 netqe12.knqe.lab.eng.bos.redhat.com systemd[1]: Starting Open vSwitch...
Sep 04 13:15:35 netqe12.knqe.lab.eng.bos.redhat.com systemd[1]: Started Open vSwitch.

[root@netqe12 tools]# ovs-vsctl show
52d30658-98c6-4015-b827-66555fe8961f
    ovs_version: "2.9.0"

[root@netqe12 tools]# rhos-release -x 14
Uninstalling all repos
cat: /var/lib/rhos-release/repos/rhos-release-rhel-7.6.repo: No such file or directory
Installed: /etc/yum.repos.d/rhos-release-rhel-7.6.repo
OSP 14 does not enable ceph repos by default
cat: /var/lib/rhos-release/repos/rhos-release-ceph-.repo: No such file or directory
Installed: /etc/yum.repos.d/rhos-release-ceph-.repo
cat: /var/lib/rhos-release/repos/rhos-release-ceph-osd-.repo: No such file or directory
Installed: /etc/yum.repos.d/rhos-release-ceph-osd-.repo
Installed: /etc/yum.repos.d/rhos-release-14.repo
# rhos-release 14   -p 2018-09-04.2

[root@netqe12 tools]# yum clean expire-cache
Loaded plugins: product-id, search-disabled-repos, subscription-manager
This system is not registered with an entitlement server. You can use subscription-manager to register.
Cleaning repos: beaker-Server beaker-Server-HighAvailability beaker-Server-NFV
              : beaker-Server-NFV-debuginfo beaker-Server-RT beaker-Server-RT-debuginfo
              : beaker-Server-ResilientStorage beaker-Server-SAP
              : beaker-Server-SAP-debuginfo beaker-Server-SAPHANA
              : beaker-Server-SAPHANA-debuginfo beaker-Server-debuginfo
              : beaker-Server-optional beaker-Server-optional-debuginfo
              : rhelosp-14.0-image-build-override rhelosp-14.0-optools-puddle
              : rhelosp-14.0-puddle
17 metadata files removed

[root@netqe12 tools]# yum provides openvswitch
Loaded plugins: product-id, search-disabled-repos, subscription-manager
This system is not registered with an entitlement server. You can use subscription-manager to register.
beaker-Server                                                      | 4.3 kB  00:00:00     
beaker-Server-HighAvailability                                     | 4.3 kB  00:00:00     
beaker-Server-NFV                                                  | 3.8 kB  00:00:00     
beaker-Server-NFV-debuginfo                                        | 3.0 kB  00:00:00     
beaker-Server-RT                                                   | 3.8 kB  00:00:00     
beaker-Server-RT-debuginfo                                         | 3.0 kB  00:00:00     
beaker-Server-ResilientStorage                                     | 4.3 kB  00:00:00     
beaker-Server-SAP                                                  | 3.8 kB  00:00:00     
beaker-Server-SAP-debuginfo                                        | 3.0 kB  00:00:00     
beaker-Server-SAPHANA                                              | 3.8 kB  00:00:00     
beaker-Server-SAPHANA-debuginfo                                    | 3.0 kB  00:00:00     
beaker-Server-debuginfo                                            | 3.0 kB  00:00:00     
beaker-Server-optional                                             | 3.8 kB  00:00:00     
beaker-Server-optional-debuginfo                                   | 3.0 kB  00:00:00     
rhelosp-14.0-image-build-override                                  | 2.9 kB  00:00:00     
rhelosp-14.0-optools-puddle                                        | 2.5 kB  00:00:00     
rhelosp-14.0-puddle                                                | 1.3 kB  00:00:00     
openvswitch-2.0.0-7.el7.x86_64 : Open vSwitch daemon/database/utilities
Repo        : beaker-Server-optional



openvswitch-2.9.0-19.el7fdp.1.x86_64 : Open vSwitch
Repo        : rhelosp-14.0-puddle



rhosp-openvswitch-2.10-0.1.el7ost.noarch : Wrapper rpm to allow installing OVS with new
                                         : versioning schemes
Repo        : rhelosp-14.0-puddle
Matched from:
Provides    : openvswitch = 2.10



openvswitch-2.9.0-56.el7fdp.x86_64 : Open vSwitch
Repo        : @rhelosp-10.0-puddle



[root@netqe12 tools]# systemctl status openvswitch
● openvswitch.service - Open vSwitch
   Loaded: loaded (/usr/lib/systemd/system/openvswitch.service; disabled; vendor preset: disabled)
   Active: active (exited) since Tue 2018-09-04 13:15:35 EDT; 2min 7s ago
  Process: 60203 ExecStart=/bin/true (code=exited, status=0/SUCCESS)
 Main PID: 60203 (code=exited, status=0/SUCCESS)

Sep 04 13:15:35 netqe12.knqe.lab.eng.bos.redhat.com systemd[1]: Starting Open vSwitch...
Sep 04 13:15:35 netqe12.knqe.lab.eng.bos.redhat.com systemd[1]: Started Open vSwitch.

[root@netqe12 tools]# yum -y update openvswitch
Loaded plugins: product-id, search-disabled-repos, subscription-manager
This system is not registered with an entitlement server. You can use subscription-manager to register.
Resolving Dependencies
--> Running transaction check
---> Package openvswitch.x86_64 0:2.9.0-56.el7fdp will be obsoleted
---> Package rhosp-openvswitch.noarch 0:2.10-0.1.el7ost will be obsoleting
--> Processing Dependency: openvswitch2.10 for package: rhosp-openvswitch-2.10-0.1.el7ost.noarch
--> Running transaction check
---> Package openvswitch2.10.x86_64 0:2.10.0-0.20180810git58a7ce6.el7fdp will be installed
--> Finished Dependency Resolution

Dependencies Resolved

==========================================================================================
 Package           Arch   Version                               Repository           Size
==========================================================================================
Installing:
 rhosp-openvswitch noarch 2.10-0.1.el7ost                       rhelosp-14.0-puddle 1.9 k
     replacing  openvswitch.x86_64 2.9.0-56.el7fdp
Installing for dependencies:
 openvswitch2.10   x86_64 2.10.0-0.20180810git58a7ce6.el7fdp    rhelosp-14.0-puddle 3.0 M

Transaction Summary
==========================================================================================
Install  1 Package (+1 Dependent package)

Total download size: 3.0 M
Downloading packages:
(1/2): rhosp-openvswitch-2.10-0.1.el7ost.noarch.rpm                | 1.9 kB  00:00:00     
(2/2): openvswitch2.10-2.10.0-0.20180810git58a7ce6.el7fdp.x86_64.r | 3.0 MB  00:00:00     
------------------------------------------------------------------------------------------
Total                                                      27 MB/s | 3.0 MB  00:00:00     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : openvswitch2.10-2.10.0-0.20180810git58a7ce6.el7fdp.x86_64              1/3 
warning: /etc/sysconfig/openvswitch created as /etc/sysconfig/openvswitch.rpmnew
  Installing : rhosp-openvswitch-2.10-0.1.el7ost.noarch                               2/3 
  Erasing    : openvswitch-2.9.0-56.el7fdp.x86_64                                     3/3 
  Verifying  : openvswitch2.10-2.10.0-0.20180810git58a7ce6.el7fdp.x86_64              1/3 
  Verifying  : rhosp-openvswitch-2.10-0.1.el7ost.noarch                               2/3 
  Verifying  : openvswitch-2.9.0-56.el7fdp.x86_64                                     3/3 

Installed:
  rhosp-openvswitch.noarch 0:2.10-0.1.el7ost                                              

Dependency Installed:
  openvswitch2.10.x86_64 0:2.10.0-0.20180810git58a7ce6.el7fdp                             

Replaced:
  openvswitch.x86_64 0:2.9.0-56.el7fdp                                                    

Complete!

[root@netqe12 tools]# rpm -qa | grep openvswitch
rhosp-openvswitch-2.10-0.1.el7ost.noarch
openvswitch2.10-2.10.0-0.20180810git58a7ce6.el7fdp.x86_64
kernel-kernel-networking-openvswitch-upgrade_ovs-1.1-5.noarch
openvswitch-selinux-extra-policy-1.0-3.el7fdp.noarch

# openvswitch service now inactive:

[root@netqe12 tools]# systemctl status openvswitch
● openvswitch.service - Open vSwitch
   Loaded: loaded (/usr/lib/systemd/system/openvswitch.service; disabled; vendor preset: disabled)
   Active: inactive (dead) since Tue 2018-09-04 13:17:52 EDT; 4s ago
 Main PID: 60203 (code=exited, status=0/SUCCESS)

Sep 04 13:15:35 netqe12.knqe.lab.eng.bos.redhat.com systemd[1]: Starting Open vSwitch...
Sep 04 13:15:35 netqe12.knqe.lab.eng.bos.redhat.com systemd[1]: Started Open vSwitch.
Sep 04 13:17:52 netqe12.knqe.lab.eng.bos.redhat.com systemd[1]: Stopping Open vSwitch...
Sep 04 13:17:52 netqe12.knqe.lab.eng.bos.redhat.com systemd[1]: Stopped Open vSwitch.

[root@netqe12 tools]# journalctl -xe
-- Unit openvswitch.service has finished shutting down.
Sep 04 13:17:52 netqe12.knqe.lab.eng.bos.redhat.com systemd[1]: Stopping Open vSwitch Forw
-- Subject: Unit ovs-vswitchd.service has begun shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit ovs-vswitchd.service has begun shutting down.
Sep 04 13:17:52 netqe12.knqe.lab.eng.bos.redhat.com yum[60377]: Erased: openvswitch-2.9.0-
Sep 04 13:17:52 netqe12.knqe.lab.eng.bos.redhat.com ovs-ctl[60421]: Exiting ovs-vswitchd (
Sep 04 13:17:52 netqe12.knqe.lab.eng.bos.redhat.com systemd[1]: Reloading.
Sep 04 13:17:52 netqe12.knqe.lab.eng.bos.redhat.com systemd[1]: Stopped Open vSwitch Forwa
-- Subject: Unit ovs-vswitchd.service has finished shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit ovs-vswitchd.service has finished shutting down.
Sep 04 13:17:52 netqe12.knqe.lab.eng.bos.redhat.com systemd[1]: Stopping Open vSwitch Data
-- Subject: Unit ovsdb-server.service has begun shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit ovsdb-server.service has begun shutting down.
Sep 04 13:17:53 netqe12.knqe.lab.eng.bos.redhat.com ovs-ctl[60462]: Exiting ovsdb-server (
Sep 04 13:17:53 netqe12.knqe.lab.eng.bos.redhat.com systemd[1]: Stopped Open vSwitch Datab
-- Subject: Unit ovsdb-server.service has finished shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit ovsdb-server.service has finished shutting down.

[root@netqe12 tools]# tail -50 /var/log/openvswitch/ovs-vswitchd.log 
2018-09-04T17:15:35.235Z|00001|vlog|INFO|opened log file /var/log/openvswitch/ovs-vswitchd.log
2018-09-04T17:15:35.239Z|00002|ovs_numa|INFO|Discovered 8 CPU cores on NUMA node 0
2018-09-04T17:15:35.239Z|00003|ovs_numa|INFO|Discovered 8 CPU cores on NUMA node 1
2018-09-04T17:15:35.239Z|00004|ovs_numa|INFO|Discovered 2 NUMA nodes and 16 CPU cores
2018-09-04T17:15:35.239Z|00005|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connecting...
2018-09-04T17:15:35.239Z|00006|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connected
2018-09-04T17:15:35.241Z|00007|dpdk|INFO|DPDK Disabled - Use other_config:dpdk-init to enable
2018-09-04T17:15:35.322Z|00008|bridge|INFO|ovs-vswitchd (Open vSwitch) 2.9.0
2018-09-04T17:17:52.750Z|00009|memory|INFO|19964 kB peak resident set size after 137.5 seconds

[root@netqe12 tools]# tail -100 /var/log/openvswitch/ovsdb-server.log 
2018-09-04T17:15:35.181Z|00001|vlog|INFO|opened log file /var/log/openvswitch/ovsdb-server.log
2018-09-04T17:15:35.187Z|00002|ovsdb_server|INFO|ovsdb-server (Open vSwitch) 2.9.0
2018-09-04T17:15:45.195Z|00003|memory|INFO|2776 kB peak resident set size after 10.0 seconds
2018-09-04T17:15:45.195Z|00004|memory|INFO|cells:16 json-caches:1 monitors:1 sessions:1

Comment 2 Flavio Leitner 2018-09-05 13:23:38 UTC

Hi,

I actually can't tell if this is a bug or a feature. :-)

I am saying that because moving from one stream to another is usually not straight forward and that's the main reason we are going with multiple streams. 

So, most probably there is a need to change/update something after the new stream is installed but before the service is started.  For example, a change in how DPDK ports are identified, or a change in a DPDK parameter.  In those cases, the Layered Product can't do it before the installation, but if the package remains with the service enabled, then most probably the service will fail during its reconfiguration.

I'd suggest that their special package that controls which OVS stream is used could do all the required changes in the %post section and then start the services when they are ready to complete the upgrade.

I'd say that it's convenient for Layered Product but we haven't discussed that.

fbl

Comment 3 Rick Alongi 2018-09-05 14:22:45 UTC

Hi,

It would be helpful to have confirmation of what we expect to happen in this scenario so that the upgrade tests can be modified accordingly, if necessary.  Right now, there is no expectation for the service to shut down after pulling down the new package.  If this is going to be the intended behavior going forward, the tests will need to be modified accordingly.  At this point, everything will likely just fail after the package update since the service is stopped.

Another thing I noticed is that the openvswitch-selinux-extra-policy package did not appear to get updated when a "yum update openvswitch" is performed; it remains at 1.0.3 (versus updating to a newer version).  Is there supposed to be a dependency to force that package to upgrade when openvswitch is updated?  Note that openvswitch-selinux-extra-policy-1.0-5.el7fdp.noarch is in the rhelosp-14.0-puddle repo (not 1.0.6 or 1.0.7).

As of now, based on this bug and BZ1625344, I can only test the upgrade to whatever is in the RHELOSP 14 repo since updating directly to an openvswitch package fails (BZ1625344).  As I mentioned, if the openvswitch service stops unexpectedly mid-upgrade, it will essentially block testing.

Please let me know whether we should consider this a blocking bug or if we should modify the tests to start the service manually.

Thanks,
Rick

Comment 4 Rick Alongi 2018-09-07 15:53:20 UTC

Just wanted to note that, because the service shuts down, performing a system reload openvswitch is no longer an option and flows have to be manually restored for traffic to resume.

Comment 5 Flavio Leitner 2018-09-12 00:53:11 UTC

I reported this in the mailing list and looks like OpenShift does not care because when they update, they update the container and so a restart is forced anyways.

The ideal flow is to do the move to the new stream using the special package and then execute actions needed to use the new stream, then start the service again. I suggest to not rebase if the downtime cannot be afforded because the risk of having issues is not trivial.

And from a package/rpm perspective, 'openvswitch' is being removed and it is expected to stop the service. The 'openvswitch2.10' is another package unrelated to 'openvswitch'.

fbl

Note You need to log in before you can comment on or make changes to this bug.