Bug 1403958
| Summary: | [fdProd] Package update from openvswitch-2.5.0-14.git20160727 to -22.git20160727 kills ovsdb-server process | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | Ihar Hrachyshka <ihrachys> | ||||
| Component: | openvswitch | Assignee: | Aaron Conole <aconole> | ||||
| Status: | CLOSED CURRENTRELEASE | QA Contact: | Rick Alongi <ralongi> | ||||
| Severity: | medium | Docs Contact: | |||||
| Priority: | high | ||||||
| Version: | 7.3 | CC: | agurenko, amuller, apevec, atragler, fbaudin, fleitner, ihrachys, ipetrova, ktraynor, lpeer, markmc, mburns, mcornea, ohochman, pablo.iranzo, pmyers, qding, rkhan, sathlang, twilson | ||||
| Target Milestone: | rc | ||||||
| Target Release: | --- | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | openvswitch-2.5.0-23.git20160727.el7fdp | Doc Type: | If docs needed, set a value | ||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | |||||||
| : | 1419632 (view as bug list) | Environment: | |||||
| Last Closed: | 2017-02-06 17:51:50 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
|
Description
Ihar Hrachyshka
2016-12-12 17:24:22 UTC
Sorry, "Note: The bug is the result of broken OSPd upgrades" should be read as "Note: The bug is the cause of broken OSPd upgrades" @Aaron, using -23 indeed fixes the update, the openvswitch is up and running. The processes are new, so it was restarted after update; I believe that's expected? I remember we were having some other problem with process restart happening in some previous package versions, that's why I am asking. Note: someone from TripleO also checks the -23 package in our upgrades scope, to see if it also solves the OSPd issue. I also suggested to test 14 to 15 to 22 package update, they may also do it after. I will ask them to report with results here. OK, the previous issue that we had with restart on package update was https://bugzilla.redhat.com/show_bug.cgi?id=1385096 I think we later worked it around for tripleo with using rpm --nopostun: https://github.com/openstack/tripleo-heat-templates/blob/stable/newton/extraconfig/tasks/major_upgrade_controller_pacemaker_2.sh#L102-L114 So maybe it indeed now makes sense to revert the patch, that's on you folks to decide. Hi,
so the rpm was installed like this by the upgrade script (confirmed by log on the platform and put there as attachment)
rpm -U --replacepkgs --nopostun ./openvswitch-2.5.0-23.git20160727.el7fdb.x86_64.rpm
And then on the working upgraded platform we had the correct package:
$ rpm -qa | grep 'openvswitch-2.5.0-23.git20160727.el7fdb.x86_64'
openvswitch-2.5.0-23.git20160727.el7fdb.x86_64
Is that incorrect ?
I don't really get the comment about getting back to bug 1385096.
Created attachment 1231765 [details]
Log of the controller upgrade.
This is the log of the controller installation where we can see the rpm installation of the openvswitch attached to the bz.
I've did successful upgrade of openvswitch-2.5.0-14.git20160727 to openvswitch-2.5.0-22.git20160727.el7fdp.bz1403958.fbl.2.x86_64. I've setup a local repo on underlcoud node and installed this repo on all nodes prior to upgrade and it was successfully picked up during upgrade procedure. Whole upgrade went very smooth with this package. Here is an output after final step of the upgrade: [stack@undercloud-0 ~]$ rpm -q openvswitch openvswitch-2.5.0-22.git20160727.el7fdp.bz1403958.fbl.2.x86_64 [stack@undercloud-0 ~]$ for i in {7..13}; do ssh heat-admin.2.$i "hostname; rpm -q openvswitch"; done ceph-0.localdomain openvswitch-2.5.0-22.git20160727.el7fdp.bz1403958.fbl.2.x86_64 compute-1.localdomain openvswitch-2.5.0-22.git20160727.el7fdp.bz1403958.fbl.2.x86_64 compute-0.localdomain openvswitch-2.5.0-22.git20160727.el7fdp.bz1403958.fbl.2.x86_64 compute-2.localdomain openvswitch-2.5.0-22.git20160727.el7fdp.bz1403958.fbl.2.x86_64 controller-1.localdomain openvswitch-2.5.0-22.git20160727.el7fdp.bz1403958.fbl.2.x86_64 controller-0.localdomain openvswitch-2.5.0-22.git20160727.el7fdp.bz1403958.fbl.2.x86_64 controller-2.localdomain openvswitch-2.5.0-22.git20160727.el7fdp.bz1403958.fbl.2.x86_64 Flavio, see comment 29, as well as an email thread "openvswitch 14 -> 22 upgrade issue" where Amit Ugol updated that your proposed OVS .rpm fixes the issue. The next step would be for the OVS team to supply this as an official build on Brew, at which point notify me and I'll contact OpenStack release delivery to bump our dependency from 14 to whatever version that ends up as. When upgrading from 2.5.0-2 to 2.6.1, because of the %postun in 2.5.0-2, openvswitch is restarted. Although updating 2.5.0-2 to fbl's 2.5.0-22 from comment 28 then updating to 2.6.1 resolves the issue, I'm not sure how we can actually be sure that people have upgraded to the latest 2.5 with the fix before updating to 2.6.1 (since postun is run from the currently installed package). Also, re: using rpm -U --nopostun from 2.5.0-2 to 2.6.1, although this doesn't restart openvswitch, it does require one to manually run systemctl daemon-reload and ovsdb-server fails to start upon the first systemctl restart openvswitch. Successive systemctl restart openvswitch calls succeed, though. Output: [terry@aio ~]$ pgrep ovsdb-server 10710 [terry@aio ~]$ sudo yum install --downloadonly --downloaddir . openvswitch ... --> Running transaction check ---> Package openvswitch.x86_64 0:2.5.0-2.el7 will be updated ---> Package openvswitch.x86_64 0:2.6.1-0.el7 will be an update --> Finished Dependency Resolution ... [terry@aio ~]$ sudo rpm -Uvh --nopostun openvswitch-2.6.1-0.el7.x86_64.rpm Preparing... ################################# [100%] Updating / installing... 1:openvswitch-2.6.1-0.el7 ################################# [ 50%] Cleaning up / removing... 2:openvswitch-2.5.0-2.el7 ################################# [100%] [terry@aio ~]$ pgrep ovsdb-server 10710 [terry@aio ~]$ sudo systemctl restart openvswitch Warning: openvswitch.service changed on disk. Run 'systemctl daemon-reload' to reload units. [terry@aio ~]$ pgrep ovsdb-server 25871 [terry@aio ~]$ sudo systemctl daemon-reload [terry@aio ~]$ pgrep ovsdb-server 25871 [terry@aio ~]$ sudo systemctl restart openvswitch [terry@aio ~]$ pgrep ovsdb-server [terry@aio ~]$ doing a stop followed by a start has identical results. Output from /var/log/messages for the restart: Jan 20 13:34:26 aio systemd: Reloading. Jan 20 13:34:26 aio systemd: [/usr/lib/systemd/system/epmd@.service:18] Failed to parse resource value, ignoring: 0 Jan 20 13:34:34 aio systemd: Stopping Open vSwitch... Jan 20 13:34:34 aio systemd: Starting Open vSwitch Database Unit... Jan 20 13:34:34 aio systemd: Starting Open vSwitch... Jan 20 13:34:34 aio systemd: Started Open vSwitch. Jan 20 13:34:34 aio ovs-ctl: ovsdb-server is already running. Jan 20 13:34:34 aio systemd: Stopping Open vSwitch... Jan 20 13:34:34 aio ovs-ctl: Enabling remote OVSDB managers [ OK ] Jan 20 13:34:34 aio systemd: Stopped Open vSwitch. Jan 20 13:34:34 aio ovs-ctl: Killing ovsdb-server (10710) [ OK ] Jan 20 13:34:34 aio systemd: Stopped Open vSwitch Database Unit. |