1511592 – l3agent can not kill keepalived process due to rootwrap filter when keepalived gets updated

Bug 1511592 - l3agent can not kill keepalived process due to rootwrap filter when keepalived gets updated

Summary: l3agent can not kill keepalived process due to rootwrap filter when keepalive...

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat OpenStack
Classification:	Red Hat
Component:	python-oslo-rootwrap
Sub Component:
Version:	10.0 (Newton)
Hardware:	Unspecified
OS:	Unspecified
Priority:	high
Severity:	high
Target Milestone:	---
Target Release:	10.0 (Newton)
Assignee:	Hervé Beraud
QA Contact:	pkomarov
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2017-11-09 16:11 UTC by Martin Schuppert
Modified:	2022-03-13 14:31 UTC (History)
CC List:	23 users (show)
Fixed In Version:	python-oslo-rootwrap-5.1.2-2.el7ost
Doc Type:	Bug Fix
Doc Text:	Previously, after a service upgrade, 'oslo.rootwrap' was unable to stop the service any more as the executable was replaced (renamed). With this update, the executables are renamed to use '/proc/$pid/cmdline' if it exists instead of '/proc/$pid/exe'. As a result, 'oslo.rootwrap' is now able to stop a service properly after an upgrade, even when the executable is replaced (renamed).
Clone Of:
Environment:
Last Closed:	2019-01-16 17:09:07 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Priority	Status	Summary	Last Updated
Launchpad	1643457	None	None	None	2017-11-15 10:31:53 UTC
OpenStack gerrit	357761	None	MERGED	Enhance _program() and _program_path()	2021-01-11 14:15:52 UTC
Red Hat Issue Tracker	OSP-4756	None	None	None	2021-12-10 15:35:00 UTC
Red Hat Knowledge Base (Solution)	3238861	None	None	None	2017-11-13 15:30:47 UTC
Red Hat Product Errata	RHBA-2019:0075	None	None	None	2019-01-16 17:09:28 UTC

Description Martin Schuppert 2017-11-09 16:11:39 UTC

Description of problem:

During OSP10 minor update from RHEL 7.3 to RHEL 7.4 keepalive gets updated.
Actions on the l3 ha instance on a controller fails with:

2017-11-09 03:25:48.447 590139 INFO neutron.agent.linux.interface [-] Device ha-aacb12f7-91 already exists
2017-11-09 03:25:48.475 590139 INFO neutron.agent.l3.ha [-] Router f284c969-79e4-4cbb-9403-7b7e6d1e490f transitioned to master
2017-11-09 03:25:48.479 590139 ERROR neutron.agent.linux.utils [-] Rootwrap error running command: ['kill', '-9', '30759']
2017-11-09 03:25:48.479 590139 ERROR neutron.agent.l3.agent [-] Error while initializing router f284c969-79e4-4cbb-9403-7b7e6d1e490f
2017-11-09 03:25:48.479 590139 ERROR neutron.agent.l3.agent Traceback (most recent call last):
2017-11-09 03:25:48.479 590139 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 353, in _router_added
2017-11-09 03:25:48.479 590139 ERROR neutron.agent.l3.agent     ri.initialize(self.process_monitor)
2017-11-09 03:25:48.479 590139 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/ha_router.py", line 116, in initialize
2017-11-09 03:25:48.479 590139 ERROR neutron.agent.l3.agent     self.update_initial_state(self.state_change_callback)
2017-11-09 03:25:48.479 590139 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/ha_router.py", line 375, in update_initial_state
2017-11-09 03:25:48.479 590139 ERROR neutron.agent.l3.agent     callback(self.router_id, state)
2017-11-09 03:25:48.479 590139 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/ha.py", line 168, in enqueue_state_change
2017-11-09 03:25:48.479 590139 ERROR neutron.agent.l3.agent     self._update_metadata_proxy(ri, router_id, state)
2017-11-09 03:25:48.479 590139 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/ha.py", line 197, in _update_metadata_proxy
2017-11-09 03:25:48.479 590139 ERROR neutron.agent.l3.agent     self.conf, router_id=ri.router_id)
2017-11-09 03:25:48.479 590139 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/metadata/driver.py", line 240, in spawn_monitored_metadata_proxy
2017-11-09 03:25:48.479 590139 ERROR neutron.agent.l3.agent     cls._migrate_python_ns_metadata_proxy_if_needed(pm)
2017-11-09 03:25:48.479 590139 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/metadata/driver.py", line 259, in _migrate_python_ns_metadata_proxy_if_needed
2017-11-09 03:25:48.479 590139 ERROR neutron.agent.l3.agent     pm.disable()
2017-11-09 03:25:48.479 590139 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/external_process.py", line 114, in disable
2017-11-09 03:25:48.479 590139 ERROR neutron.agent.l3.agent     utils.execute(cmd, run_as_root=True)
2017-11-09 03:25:48.479 590139 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 124, in execute
2017-11-09 03:25:48.479 590139 ERROR neutron.agent.l3.agent     execute_rootwrap_daemon(cmd, process_input, addl_env))
2017-11-09 03:25:48.479 590139 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 111, in execute_rootwrap_daemon
2017-11-09 03:25:48.479 590139 ERROR neutron.agent.l3.agent     LOG.error(_LE("Rootwrap error running command: %s"), cmd)
2017-11-09 03:25:48.479 590139 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-11-09 03:25:48.479 590139 ERROR neutron.agent.l3.agent     self.force_reraise()
2017-11-09 03:25:48.479 590139 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-11-09 03:25:48.479 590139 ERROR neutron.agent.l3.agent     six.reraise(self.type_, self.value, self.tb)
2017-11-09 03:25:48.479 590139 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 108, in execute_rootwrap_daemon
2017-11-09 03:25:48.479 590139 ERROR neutron.agent.l3.agent     return client.execute(cmd, process_input)
2017-11-09 03:25:48.479 590139 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/oslo_rootwrap/client.py", line 129, in execute
2017-11-09 03:25:48.479 590139 ERROR neutron.agent.l3.agent     res = proxy.run_one_command(cmd, stdin)
2017-11-09 03:25:48.479 590139 ERROR neutron.agent.l3.agent   File "<string>", line 2, in run_one_command
2017-11-09 03:25:48.479 590139 ERROR neutron.agent.l3.agent   File "/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in _callmethod
2017-11-09 03:25:48.479 590139 ERROR neutron.agent.l3.agent     raise convert_to_error(kind, result)
2017-11-09 03:25:48.479 590139 ERROR neutron.agent.l3.agent RemoteError: 
2017-11-09 03:25:48.479 590139 ERROR neutron.agent.l3.agent ---------------------------------------------------------------------------
2017-11-09 03:25:48.479 590139 ERROR neutron.agent.l3.agent Unserializable message: ('#ERROR', ValueError('I/O operation on closed file',))
2017-11-09 03:25:48.479 590139 ERROR neutron.agent.l3.agent ---------------------------------------------------------------------------

In /usr/share/neutron/rootwrap/l3.filters we have this KillFilter for keepalived:
~~~
kill_keepalived: KillFilter, root, /usr/sbin/keepalived, -HUP, -15, -9
~~~

This is not correct for the running process as the previous binary for the process is deleted/replaced by the new one:

[root@controller-2 heat-admin]# ll /proc/697659/exe 
lrwxrwxrwx. 1 root root 0 Nov  9 13:20 /proc/697659/exe -> /usr/sbin/keepalived;5a0456a9 (deleted)

[root@controller-2 heat-admin]# ps -ef |grep 697659
root      697659       1  0 13:20 ?        00:00:00 keepalived -P -f /var/lib/neutron/ha_confs/36e6d039-0799-4e3e-a8c1-e5168760a22f/keepalived.conf -p /var/lib/neutron/ha_confs/36e6d039-0799-4e3e-a8c1-e5168760a22f.pid -r /var/lib/neutron/ha_confs/36e6d039-0799-4e3e-a8c1-e5168760a22f.pid-vrrp
root      697660  697659  0 13:20 ?        00:00:00 keepalived -P -f /var/lib/neutron/ha_confs/36e6d039-0799-4e3e-a8c1-e5168760a22f/keepalived.conf -p /var/lib/neutron/ha_confs/36e6d039-0799-4e3e-a8c1-e5168760a22f.pid -r /var/lib/neutron/ha_confs/36e6d039-0799-4e3e-a8c1-e5168760a22f.pid-vrrp
root      742325  594552  0 13:26 pts/0    00:00:00 grep --color=auto 697659

Version-Release number of selected component (if applicable):
seen during OSP10 minor upgrade from RHEL 7.3 to RHEL 7.4, 
but should apply to other versions as well. I have not checked
the filter
keepalived-1.2.13-9.el7_3 -> keepalived-1.3.5-1.el7
openstack-neutron-9.3.1-2 -> openstack-neutron-9.4.1-4

How reproducible:
always with the bellow steps, which are likely not the same which happen during an 
update, but we can trigger it like this.


Steps to Reproduce:
0) start with
keepalived-1.3.5-1.el7.x86_64

1) create a HA router
[stack@undercloud-0 ~]$ neutron l3-agent-list-hosting-router tr
+--------------------------------------+--------------------------+----------------+-------+----------+
| id                                   | host                     | admin_state_up | alive | ha_state |
+--------------------------------------+--------------------------+----------------+-------+----------+
| dc6b4425-be2d-4896-9ed1-0fd4af4e5b55 | controller-0.localdomain | True           | :-)   | active   |
| e8c6fc98-6e43-4609-87d5-10b834cbbed5 | controller-1.localdomain | True           | :-)   | standby  |
| 69fdd8f9-6dbe-466c-8a1d-b03e3de3a5ae | controller-2.localdomain | True           | :-)   | standby  |
+--------------------------------------+--------------------------+----------------+-------+----------+

2) downgrade or upgrade keepalive, here we downgrade controller-2
[root@controller-2 heat-admin]# yum downgrade keepalived-1.2.13-9.el7_3.x86_64.rpm

3) remove router from agent
[stack@undercloud-0 ~]$ neutron l3-agent-router-remove  69fdd8f9-6dbe-466c-8a1d-b03e3de3a5ae tr
Removed router tr from L3 agent

[stack@undercloud-0 ~]$ neutron l3-agent-list-hosting-router tr
+--------------------------------------+--------------------------+----------------+-------+----------+
| id                                   | host                     | admin_state_up | alive | ha_state |
+--------------------------------------+--------------------------+----------------+-------+----------+
| dc6b4425-be2d-4896-9ed1-0fd4af4e5b55 | controller-0.localdomain | True           | :-)   | active   |
| e8c6fc98-6e43-4609-87d5-10b834cbbed5 | controller-1.localdomain | True           | :-)   | standby  |
+--------------------------------------+--------------------------+----------------+-------+----------+

In l3 agent log:
2017-11-09 13:23:52.422 667494 DEBUG neutron.agent.linux.utils [-] Running command (rootwrap daemon): ['kill', '-15', '697659'] execute_rootwrap_daemon /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:105
2017-11-09 13:23:52.426 667494 ERROR neutron.agent.linux.utils [-] Rootwrap error running command: ['kill', '-15', '697659']
2017-11-09 13:23:52.427 667494 ERROR neutron.agent.l3.agent [-] Error while deleting router 36e6d039-0799-4e3e-a8c1-e5168760a22f
2017-11-09 13:23:52.427 667494 ERROR neutron.agent.l3.agent Traceback (most recent call last):
2017-11-09 13:23:52.427 667494 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 370, in _safe_router_removed
2017-11-09 13:23:52.427 667494 ERROR neutron.agent.l3.agent     self._router_removed(router_id)
2017-11-09 13:23:52.427 667494 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 389, in _router_removed
2017-11-09 13:23:52.427 667494 ERROR neutron.agent.l3.agent     ri.delete(self)
2017-11-09 13:23:52.427 667494 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/ha_router.py", line 422, in delete
2017-11-09 13:23:52.427 667494 ERROR neutron.agent.l3.agent     self.disable_keepalived()
2017-11-09 13:23:52.427 667494 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/ha_router.py", line 159, in disable_keepalived
2017-11-09 13:23:52.427 667494 ERROR neutron.agent.l3.agent     self.keepalived_manager.disable()
2017-11-09 13:23:52.427 667494 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/keepalived.py", line 451, in disable
2017-11-09 13:23:52.427 667494 ERROR neutron.agent.l3.agent     pm.disable(sig='15')
2017-11-09 13:23:52.427 667494 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/external_process.py", line 114, in disable
2017-11-09 13:23:52.427 667494 ERROR neutron.agent.l3.agent     utils.execute(cmd, run_as_root=True)
2017-11-09 13:23:52.427 667494 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 124, in execute
2017-11-09 13:23:52.427 667494 ERROR neutron.agent.l3.agent     execute_rootwrap_daemon(cmd, process_input, addl_env))
2017-11-09 13:23:52.427 667494 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 111, in execute_rootwrap_daemon
2017-11-09 13:23:52.427 667494 ERROR neutron.agent.l3.agent     LOG.error(_LE("Rootwrap error running command: %s"), cmd)
2017-11-09 13:23:52.427 667494 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-11-09 13:23:52.427 667494 ERROR neutron.agent.l3.agent     self.force_reraise()
2017-11-09 13:23:52.427 667494 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-11-09 13:23:52.427 667494 ERROR neutron.agent.l3.agent     six.reraise(self.type_, self.value, self.tb)
2017-11-09 13:23:52.427 667494 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 108, in execute_rootwrap_daemon
2017-11-09 13:23:52.427 667494 ERROR neutron.agent.l3.agent     return client.execute(cmd, process_input)
2017-11-09 13:23:52.427 667494 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/oslo_rootwrap/client.py", line 129, in execute
2017-11-09 13:23:52.427 667494 ERROR neutron.agent.l3.agent     res = proxy.run_one_command(cmd, stdin)
2017-11-09 13:23:52.427 667494 ERROR neutron.agent.l3.agent   File "<string>", line 2, in run_one_command
2017-11-09 13:23:52.427 667494 ERROR neutron.agent.l3.agent   File "/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in _callmethod
2017-11-09 13:23:52.427 667494 ERROR neutron.agent.l3.agent     raise convert_to_error(kind, result)
2017-11-09 13:23:52.427 667494 ERROR neutron.agent.l3.agent RemoteError: 
2017-11-09 13:23:52.427 667494 ERROR neutron.agent.l3.agent ---------------------------------------------------------------------------
2017-11-09 13:23:52.427 667494 ERROR neutron.agent.l3.agent Unserializable message: ('#ERROR', ValueError('I/O operation on closed file',))
2017-11-09 13:23:52.427 667494 ERROR neutron.agent.l3.agent ---------------------------------------------------------------------------
2017-11-09 13:23:52.427 667494 ERROR neutron.agent.l3.agent 


[root@controller-2 heat-admin]# ll /proc/697659/exe 
lrwxrwxrwx. 1 root root 0 Nov  9 13:20 /proc/697659/exe -> /usr/sbin/keepalived;5a0456a9 (deleted)

[root@controller-2 heat-admin]# ps -ef |grep 697659
root      697659       1  0 13:20 ?        00:00:00 keepalived -P -f /var/lib/neutron/ha_confs/36e6d039-0799-4e3e-a8c1-e5168760a22f/keepalived.conf -p /var/lib/neutron/ha_confs/36e6d039-0799-4e3e-a8c1-e5168760a22f.pid -r /var/lib/neutron/ha_confs/36e6d039-0799-4e3e-a8c1-e5168760a22f.pid-vrrp
root      697660  697659  0 13:20 ?        00:00:00 keepalived -P -f /var/lib/neutron/ha_confs/36e6d039-0799-4e3e-a8c1-e5168760a22f/keepalived.conf -p /var/lib/neutron/ha_confs/36e6d039-0799-4e3e-a8c1-e5168760a22f.pid -r /var/lib/neutron/ha_confs/36e6d039-0799-4e3e-a8c1-e5168760a22f.pid-vrrp
root      742325  594552  0 13:26 pts/0    00:00:00 grep --color=auto 697659

In /usr/share/neutron/rootwrap/l3.filters we have this KillFilter for keepalived:
~~~
kill_keepalived: KillFilter, root, /usr/sbin/keepalived, -HUP, -15, -9
~~~

Actual results:
managing the ha router is not possible on the agent and result in :
2017-11-09 13:23:52.427 667494 ERROR neutron.agent.l3.agent Unserializable message: ('#ERROR', ValueError('I/O operation on closed file',))

Expected results:
keepalived process can be killed by l3agent after an update.

Additional info:

We can fix this by creating an additional KillFilter for keepalived:

# mkdir -p /etc/neutron/rootwrap.d

Create /etc/neutron/rootwrap.d/l3agent2.filter with the following content:
~~~
[Filters]
# Keepalived
kill_keepalived2: KillFilter, root, keepalived, -HUP, -15, -9
~~~

Test:

[root@controller-2 heat-admin]# ps -ef |grep keep
root        2310       1  0 Oct23 ttyS0    00:00:00 /sbin/agetty --keep-baud 115200 38400 9600 ttyS0 vt220
neutron   845675       1  0 13:38 ?        00:00:00 /usr/bin/python2 /bin/neutron-keepalived-state-change --router_id=36e6d039-0799-4e3e-a8c1-e5168760a22f --namespace=qrouter-36e6d039-0799-4e3e-a8c1-e5168760a22f --conf_dir=/var/lib/neutron/ha_confs/36e6d039-0799-4e3e-a8c1-e5168760a22f --monitor_interface=ha-d044e9cc-14 --monitor_cidr=169.254.0.1/24 --pid_file=/var/lib/neutron/external/pids/36e6d039-0799-4e3e-a8c1-e5168760a22f.monitor.pid --state_path=/var/lib/neutron --user=997 --group=994
root      845845       1  0 13:38 ?        00:00:00 keepalived -P -f /var/lib/neutron/ha_confs/36e6d039-0799-4e3e-a8c1-e5168760a22f/keepalived.conf -p /var/lib/neutron/ha_confs/36e6d039-0799-4e3e-a8c1-e5168760a22f.pid -r /var/lib/neutron/ha_confs/36e6d039-0799-4e3e-a8c1-e5168760a22f.pid-vrrp
root      845846  845845  0 13:38 ?        00:00:00 keepalived -P -f /var/lib/neutron/ha_confs/36e6d039-0799-4e3e-a8c1-e5168760a22f/keepalived.conf -p /var/lib/neutron/ha_confs/36e6d039-0799-4e3e-a8c1-e5168760a22f.pid -r /var/lib/neutron/ha_confs/36e6d039-0799-4e3e-a8c1-e5168760a22f.pid-vrrp

# ll /proc/845845/exe 
lrwxrwxrwx. 1 root root 0 Nov  9 13:38 /proc/845845/exe -> /usr/sbin/keepalived

# rpm -qa |grep keepalive
keepalived-1.2.13-9.el7_3.x86_64

# ll /proc/845845/exe 
lrwxrwxrwx. 1 root root 0 Nov  9 13:38 /proc/845845/exe -> /usr/sbin/keepalived;5a045bc2 (deleted)

Now when we remove the agent it gets killed properly:
2017-11-09 13:45:03.672 835191 DEBUG neutron.agent.linux.utils [-] Running command (rootwrap daemon): ['kill', '-15', '845845'] execute_rootwrap_daemon /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:105
2017-11-09 13:45:03.690 835191 DEBUG neutron.agent.linux.utils [-] Exit code: 0 execute /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:150

This is similar to BZ 1470975 , which has also be seen during the minor update.

Comment 2 Jakub Libosvar 2017-11-13 14:41:54 UTC

Ihar found a fix in upstream that was merged to Ocata. We probably need a backport to Newton.

Comment 3 Martin Schuppert 2017-11-13 14:48:07 UTC

(In reply to Jakub Libosvar from comment #2)
> Ihar found a fix in upstream that was merged to Ocata. We probably need a
> backport to Newton.

How about older releases? It seems we have the same filter e.g. in OSP9

Comment 4 Jakub Libosvar 2017-11-13 15:03:01 UTC

(In reply to Martin Schuppert from comment #3)
> (In reply to Jakub Libosvar from comment #2)
> > Ihar found a fix in upstream that was merged to Ocata. We probably need a
> > backport to Newton.
> 
> How about older releases? It seems we have the same filter e.g. in OSP9

Good point. After we confirm we have a fix, we'll need to clone this BZ to all other older releases.

Comment 9 Pablo Iranzo Gómez 2018-07-23 07:16:14 UTC

(In reply to Jakub Libosvar from comment #2)
> Ihar found a fix in upstream that was merged to Ocata. We probably need a
> backport to Newton.

Ihar, which is the one merged in Ocata? the one linked to this bz is not yet merged

Comment 16 Hervé Beraud 2018-12-13 09:59:16 UTC

Hey James,

Doc text added
Best

Comment 20 pkomarov 2018-12-17 08:06:39 UTC

Verified, 

[stack@undercloud-0 ~]$ rhos-release -L
Installed repositories (rhel-7.6):
  10
  ceph-2
  ceph-osd-2
  rhel-7.6

[stack@undercloud-0 ~]$ cat core_puddle_version 
2018-12-13.1[stack@undercloud-0 ~]$ 

openstack  router create --disable tr
openstack router set --enable tr

[stack@undercloud-0 ~]$  neutron l3-agent-list-hosting-router tr
+--------------------------------------+--------------------------+----------------+-------+----------+
| id                                   | host                     | admin_state_up | alive | ha_state |
+--------------------------------------+--------------------------+----------------+-------+----------+
| 822898a6-97be-4284-a441-024e2ef5aad6 | controller-1.localdomain | True           | :-)   | active   |
| bd199043-84eb-45fe-a103-32404b1c1426 | controller-0.localdomain | True           | :-)   | standby  |
+--------------------------------------+--------------------------+----------------+-------+----------+


#downgrade:
yum downgrade ./keepalived-1.2.13-9.el7_3.1.x86_64.rpm 

Removed:
  keepalived.x86_64 0:1.3.5-5.el7                                                                         

Installed:
  keepalived.x86_64 0:1.2.13-9.el7_3.1                                                                    



[stack@undercloud-0 ~]$ neutron l3-agent-list-hosting-router tr
+--------------------------------------+--------------------------+----------------+-------+----------+
| id                                   | host                     | admin_state_up | alive | ha_state |
+--------------------------------------+--------------------------+----------------+-------+----------+
| 822898a6-97be-4284-a441-024e2ef5aad6 | controller-1.localdomain | True           | :-)   | active   |
| bd199043-84eb-45fe-a103-32404b1c1426 | controller-0.localdomain | True           | :-)   | standby  |
+--------------------------------------+--------------------------+----------------+-------+----------+
[stack@undercloud-0 ~]$  neutron l3-agent-router-remove 822898a6-97be-4284-a441-024e2ef5aad6 tr
Removed router tr from L3 agent




#monitorring of l3agent logs while router was removed shows no errors: 

2018-12-17 07:59:20.213 128617 DEBUG oslo_service.periodic_task [req-45968e44-b112-46fc-8810-8c6c1e837155 - - - - -] Running periodic task L3NATAgentWithStateReport.periodic_sync_routers_task run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215
2018-12-17 07:59:33.487 128617 DEBUG oslo_messaging._drivers.amqpdriver [-] CALL msg_id: ef0e60e393c941a480f1290a6609309a exchange 'neutron' topic 'q-reports-plugin' _send /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:568
2018-12-17 07:59:33.493 128617 DEBUG oslo_concurrency.lockutils [-] Lock "_check_child_processes" acquired by "neutron.agent.linux.external_process._check_child_processes" :: waited 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:270
2018-12-17 07:59:33.494 128617 DEBUG oslo_concurrency.lockutils [-] Lock "_check_child_processes" released by "neutron.agent.linux.external_process._check_child_processes" :: held 0.001s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:282
2018-12-17 07:59:33.495 128617 DEBUG oslo_concurrency.lockutils [-] Lock "_check_child_processes" acquired by "neutron.agent.linux.external_process._check_child_processes" :: waited 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:270
2018-12-17 07:59:33.495 128617 DEBUG oslo_concurrency.lockutils [-] Lock "_check_child_processes" released by "neutron.agent.linux.external_process._check_child_processes" :: held 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:282
2018-12-17 07:59:33.512 128617 DEBUG oslo_messaging._drivers.amqpdriver [-] received reply msg_id: ef0e60e393c941a480f1290a6609309a __call__ /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:416
2018-12-17 08:00:00.214 128617 DEBUG oslo_service.periodic_task [req-45968e44-b112-46fc-8810-8c6c1e837155 - - - - -] Running periodic task L3NATAgentWithStateReport.periodic_sync_routers_task run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215
2018-12-17 08:00:03.485 128617 DEBUG oslo_messaging._drivers.amqpdriver [-] CALL msg_id: 0ee52e48984940a48124351c3b8d1e51 exchange 'neutron' topic 'q-reports-plugin' _send /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:568
2018-12-17 08:00:03.511 128617 DEBUG oslo_messaging._drivers.amqpdriver [-] received reply msg_id: 0ee52e48984940a48124351c3b8d1e51 __call__ /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:416
2018-12-17 08:00:33.483 128617 DEBUG oslo_messaging._drivers.amqpdriver [-] CALL msg_id: fa7e084e06a44ce0bcca74876c346f12 exchange 'neutron' topic 'q-reports-plugin' _send /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:568
2018-12-17 08:00:33.493 128617 DEBUG oslo_concurrency.lockutils [-] Lock "_check_child_processes" acquired by "neutron.agent.linux.external_process._check_child_processes" :: waited 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:270
2018-12-17 08:00:33.494 128617 DEBUG oslo_concurrency.lockutils [-] Lock "_check_child_processes" released by "neutron.agent.linux.external_process._check_child_processes" :: held 0.002s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:282
2018-12-17 08:00:33.495 128617 DEBUG oslo_concurrency.lockutils [-] Lock "_check_child_processes" acquired by "neutron.agent.linux.external_process._check_child_processes" :: waited 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:270
2018-12-17 08:00:33.495 128617 DEBUG oslo_concurrency.lockutils [-] Lock "_check_child_processes" released by "neutron.agent.linux.external_process._check_child_processes" :: held 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:282
2018-12-17 08:00:33.514 128617 DEBUG oslo_messaging._drivers.amqpdriver [-] received reply msg_id: fa7e084e06a44ce0bcca74876c346f12 __call__ /usr/lib/python2.7/site-packages/oslo_m

Comment 27 errata-xmlrpc 2019-01-16 17:09:07 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0075

Note You need to log in before you can comment on or make changes to this bug.

amuller
apevec
bperkins
chjones
chrisw
dabarzil
fdinitto
gkadam
hberaud
ihrachys
jamsmith
jlibosva
lhh
michele
mschuppe
nyechiel
ojanas
pablo.iranzo
pkomarov
sandyada
srevivo
ushkalim
vstinner