2080966 – failed to reload listeners because of selinux error

Bug 2080966 - failed to reload listeners because of selinux error

Summary: failed to reload listeners because of selinux error

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat OpenStack
Classification:	Red Hat
Component:	openstack-octavia
Sub Component:
Version:	17.0 (Wallaby)
Hardware:	Unspecified
OS:	Unspecified
Priority:	high
Severity:	high
Target Milestone:	Alpha
Target Release:	17.0
Assignee:	Gregory Thiemonge
QA Contact:	Bruna Bonguardo
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2022-05-02 13:25 UTC by Gregory Thiemonge
Modified:	2022-09-21 12:21 UTC (History)
CC List:	11 users (show)
Fixed In Version:	openstack-octavia-8.0.2-0.20220506160857.3b1068e.el9ost openstack-selinux-0.8.31-0.20220428171339.400a9a5.el9ost
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2022-09-21 12:20:53 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Priority	Status	Summary	Last Updated
Github	redhat-openstack openstack-selinux pull 90	None	Merged	Octavia: add new policies for the Amphora	2022-05-04 10:15:55 UTC
OpenStack gerrit	840315	None	NEW	Apply openstack-selinux policies in Centos amphorae	2022-05-03 14:22:18 UTC
Red Hat Issue Tracker	OSP-14976	None	None	None	2022-05-02 13:40:24 UTC
Red Hat Product Errata	RHEA-2022:6543	None	None	None	2022-09-21 12:21:05 UTC

Description Gregory Thiemonge 2022-05-02 13:25:43 UTC

Description of problem:
Tests are failing in OSP17 with Octavia, Octavia cannot reload the listeners in the amphora

CI logs (worker.log):

2022-04-28 18:33:41.348 18 DEBUG octavia.amphorae.drivers.haproxy.rest_api_driver [req-cb04087e-5400-49d8-a2bc-7c66c13d8e9c - 3c1743617d4142c5bd0bf77332d9fab3 - - -] request url loadbalancer/047b4ae5-739a-42c7-9ebe-7e42e53d4f20/03c9c508-51c3-4419-a346-436880921e9b/haproxy request /usr/lib/python3.9/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py:678
2022-04-28 18:33:41.349 18 DEBUG octavia.amphorae.drivers.haproxy.rest_api_driver [req-cb04087e-5400-49d8-a2bc-7c66c13d8e9c - 3c1743617d4142c5bd0bf77332d9fab3 - - -] request url https://172.24.1.172:9443/1.0/loadbalancer/047b4ae5-739a-42c7-9ebe-7e42e53d4f20/03c9c508-51c3-4419-a346-436880921e9b/haproxy request /usr/lib/python3.9/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py:681
2022-04-28 18:33:41.612 14 DEBUG octavia.amphorae.drivers.haproxy.rest_api_driver [req-4ab46e95-6a98-4456-8569-8240c7ea7b57 - aaaf8e8845e84605aedb8c56ab5aff3f - - -] Connected to amphora. Response: <Response [500]> request /usr/lib/python3.9/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py:701
2022-04-28 18:33:41.612 14 ERROR octavia.amphorae.drivers.haproxy.exceptions [req-4ab46e95-6a98-4456-8569-8240c7ea7b57 - aaaf8e8845e84605aedb8c56ab5aff3f - - -] Amphora agent returned unexpected result code 500 with response {'message': 'Error starting haproxy', 'details': 'Redirecting to /bin/systemctl start haproxy-56dd3d4b-9649-49ee-a554-b2b15339e6a3.service\nJob for haproxy-56dd3d4b-9649-49ee-a554-b2b15339e6a3.service failed because the control process exited with error code.\nSee "systemctl status haproxy-56dd3d4b-9649-49ee-a554-b2b15339e6a3.service" and "journalctl -xeu haproxy-56dd3d4b-9649-49ee-a554-b2b15339e6a3.service" for details.\n'}
2022-04-28 18:33:41.619 14 WARNING octavia.controller.worker.v1.controller_worker [-] Task 'octavia.controller.worker.v1.tasks.amphora_driver_tasks.ListenersUpdate' (42c4a026-725a-4a11-933e-70a9ed5fbb18) transitioned into state 'FAILURE' from state 'RUNNING'
3 predecessors (most recent first):
  Atom 'octavia.controller.worker.v1.tasks.database_tasks.MarkPoolPendingCreateInDB' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'pool': <octavia.common.data_models.Pool object at 0x7f7b0c38f940>}, 'provides': None}
  |__Atom 'octavia.controller.worker.v1.tasks.lifecycle_tasks.PoolToErrorOnRevertTask' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'pool': <octavia.common.data_models.Pool object at 0x7f7b0c38f940>, 'listeners': [], 'loadbalancer': <octavia.common.data_models.LoadBalancer object at 0x7f7b0c3514c0>}, 'provides': None}
     |__Flow 'octavia-create-pool-flow': octavia.amphorae.drivers.haproxy.exceptions.InternalServerError: Internal Server Error
2022-04-28 18:33:41.619 14 ERROR octavia.controller.worker.v1.controller_worker Traceback (most recent call last):
2022-04-28 18:33:41.619 14 ERROR octavia.controller.worker.v1.controller_worker   File "/usr/lib/python3.9/site-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task
2022-04-28 18:33:41.619 14 ERROR octavia.controller.worker.v1.controller_worker     result = task.execute(**arguments)
2022-04-28 18:33:41.619 14 ERROR octavia.controller.worker.v1.controller_worker   File "/usr/lib/python3.9/site-packages/octavia/controller/worker/v1/tasks/amphora_driver_tasks.py", line 100, in execute
2022-04-28 18:33:41.619 14 ERROR octavia.controller.worker.v1.controller_worker     self.amphora_driver.update(loadbalancer)
2022-04-28 18:33:41.619 14 ERROR octavia.controller.worker.v1.controller_worker   File "/usr/lib/python3.9/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py", line 253, in update
2022-04-28 18:33:41.619 14 ERROR octavia.controller.worker.v1.controller_worker     self.update_amphora_listeners(loadbalancer, amphora)
2022-04-28 18:33:41.619 14 ERROR octavia.controller.worker.v1.controller_worker   File "/usr/lib/python3.9/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py", line 225, in update_amphora_listeners
2022-04-28 18:33:41.619 14 ERROR octavia.controller.worker.v1.controller_worker     self.clients[amphora.api_version].reload_listener(
2022-04-28 18:33:41.619 14 ERROR octavia.controller.worker.v1.controller_worker   File "/usr/lib/python3.9/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py", line 907, in _action
2022-04-28 18:33:41.619 14 ERROR octavia.controller.worker.v1.controller_worker     return exc.check_exception(r)
2022-04-28 18:33:41.619 14 ERROR octavia.controller.worker.v1.controller_worker   File "/usr/lib/python3.9/site-packages/octavia/amphorae/drivers/haproxy/exceptions.py", line 44, in check_exception
2022-04-28 18:33:41.619 14 ERROR octavia.controller.worker.v1.controller_worker     raise responses[status_code]()
2022-04-28 18:33:41.619 14 ERROR octavia.controller.worker.v1.controller_worker octavia.amphorae.drivers.haproxy.exceptions.InternalServerError: Internal Server Error


Another run (logs from the amphora)

May  2 14:18:36 amphora-a44a1007-6fe5-4213-88f8-a21e8a10c5f3 systemd[1]: Starting HAProxy Load Balancer...                                                                                                                                    
May  2 14:18:36 amphora-a44a1007-6fe5-4213-88f8-a21e8a10c5f3 ip[1163]: Cannot open network namespace "amphora-haproxy": Permission denied                                                                                                      
May  2 14:18:36 amphora-a44a1007-6fe5-4213-88f8-a21e8a10c5f3 systemd[1]: haproxy-a26aec91-a004-4d98-b84e-0499b286ed48.service: Main process exited, code=exited, status=255/EXCEPTION                                                          
May  2 14:18:36 amphora-a44a1007-6fe5-4213-88f8-a21e8a10c5f3 systemd[1]: haproxy-a26aec91-a004-4d98-b84e-0499b286ed48.service: Failed with result 'exit-code'.                                                                                
May  2 14:18:36 amphora-a44a1007-6fe5-4213-88f8-a21e8a10c5f3 systemd[1]: Failed to start HAProxy Load Balancer.


SELinux related logs with permissive=1:

type=AVC msg=audit(1651494369.458:466): avc:  denied  { read } for  pid=1705 comm="ip" dev="nsfs" ino=4026532274 scontext=system_u:system_r:haproxy_t:s0 tcontext=system_u:object_r:nsfs_t:s0 tclass=file permissive=1
type=AVC msg=audit(1651494369.458:466): avc:  denied  { open } for  pid=1705 comm="ip" path="/run/netns/amphora-haproxy" dev="nsfs" ino=4026532274 scontext=system_u:system_r:haproxy_t:s0 tcontext=system_u:object_r:nsfs_t:s0 tclass=file permissive=1


Version-Release number of selected component (if applicable):
17.0

Comment 13 Omer Schwartz 2022-05-19 09:26:54 UTC

The Octavia OSP17 jobs run on RHEL9, so as the following build

https://rhos-ci-jenkins.lab.eng.tlv2.redhat.com/view/DFG/view/network/view/octavia/job/DFG-network-octavia-17.0_director-rhel-virthost-3cont_3comp-ipv4-geneve-actstby/45/

contains tests which show that the fix works, I am moving this BZ to VERIFIED.

Comment 18 errata-xmlrpc 2022-09-21 12:20:53 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Release of components for Red Hat OpenStack Platform 17.0 (Wallaby)), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2022:6543

Note You need to log in before you can comment on or make changes to this bug.