2078539 – Octavia cannot reload haproxy because of selinux policies

Bug 2078539 - Octavia cannot reload haproxy because of selinux policies

Summary: Octavia cannot reload haproxy because of selinux policies

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat OpenStack
Classification:	Red Hat
Component:	openstack-octavia
Sub Component:
Version:	16.2 (Train)
Hardware:	Unspecified
OS:	Unspecified
Priority:	high
Severity:	high
Target Milestone:	z3
Target Release:	16.2 (Train on RHEL 8.4)
Assignee:	Gregory Thiemonge
QA Contact:	Bruna Bonguardo
Docs Contact:
URL:
Whiteboard:
Depends On:	2073491
Blocks:	2136558
TreeView+	depends on / blocked

Reported:	2022-04-25 14:31 UTC by Gregory Thiemonge
Modified:	2022-10-20 16:12 UTC (History)
CC List:	14 users (show)
Fixed In Version:	openstack-octavia-5.1.3-2.20220328185156.el8ost
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:	2073491
Clones:	2136558 (view as bug list)
Environment:
Last Closed:	2022-06-22 16:06:30 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Priority	Status	Summary	Last Updated
OpenStack gerrit	838741	None	MERGED	Save the HAProxy state outside of its systemd unit	2022-04-25 14:36:06 UTC
Red Hat Issue Tracker	OSP-14857	None	None	None	2022-04-25 14:56:07 UTC
Red Hat Product Errata	RHBA-2022:4793	None	None	None	2022-06-22 16:07:00 UTC

Description Gregory Thiemonge 2022-04-25 14:31:42 UTC

+++ This bug was initially created as a clone of Bug #2073491 +++

Description of problem:

In OSP17 (RHEL8 and 9), Octavia fails to reload haproxy after each configuration update.

The worker logs show:

2022-04-08 13:45:16.578 38 DEBUG octavia.controller.worker.v1.controller_worker [-] Task 'octavia.controller.worker.v1.tasks.amphora_driver_tasks.ListenersUpdate' (66ff05b8-0756-4e4d-85df-f09a71805b4b) transitioned into state 'RUNNING' from state 'PENDING' _task_receiver /usr/lib/python3.6/site-packages/taskflow/listeners/logging.py:192
2022-04-08 13:45:16.578 38 DEBUG octavia.amphorae.drivers.haproxy.rest_api_driver [req-627fe0bb-cbb1-491e-a682-ccf199cb577f - 2886febc2f0c44fea2250ec811834f37 - - -] request url / request /usr/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py:678
2022-04-08 13:45:16.579 38 DEBUG octavia.amphorae.drivers.haproxy.rest_api_driver [req-627fe0bb-cbb1-491e-a682-ccf199cb577f - 2886febc2f0c44fea2250ec811834f37 - - -] request url https://172.24.3.163:9443// request /usr/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py:681
2022-04-08 13:45:16.597 38 DEBUG octavia.amphorae.drivers.haproxy.rest_api_driver [req-627fe0bb-cbb1-491e-a682-ccf199cb577f - 2886febc2f0c44fea2250ec811834f37 - - -] Connected to amphora. Response: <Response [200]> request /usr/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py:702
2022-04-08 13:45:16.597 38 DEBUG octavia.amphorae.drivers.haproxy.rest_api_driver [req-627fe0bb-cbb1-491e-a682-ccf199cb577f - 2886febc2f0c44fea2250ec811834f37 - - -] Amphora 3d8868f5-088f-44e7-88b4-fe860f2f0972 has API version 1.0 _populate_amphora_api_version /usr/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py:112
2022-04-08 13:45:16.598 38 DEBUG octavia.amphorae.drivers.haproxy.rest_api_driver [req-627fe0bb-cbb1-491e-a682-ccf199cb577f - 2886febc2f0c44fea2250ec811834f37 - - -] request url info request /usr/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py:678
2022-04-08 13:45:16.598 38 DEBUG octavia.amphorae.drivers.haproxy.rest_api_driver [req-627fe0bb-cbb1-491e-a682-ccf199cb577f - 2886febc2f0c44fea2250ec811834f37 - - -] request url https://172.24.3.163:9443/1.0/info request /usr/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py:681
2022-04-08 13:45:16.663 38 DEBUG octavia.amphorae.drivers.haproxy.rest_api_driver [req-627fe0bb-cbb1-491e-a682-ccf199cb577f - 2886febc2f0c44fea2250ec811834f37 - - -] Connected to amphora. Response: <Response [200]> request /usr/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py:702
2022-04-08 13:45:16.663 38 DEBUG octavia.amphorae.drivers.haproxy.rest_api_driver [req-627fe0bb-cbb1-491e-a682-ccf199cb577f - 2886febc2f0c44fea2250ec811834f37 - - -] Amphora 3d8868f5-088f-44e7-88b4-fe860f2f0972 has API version 1.0 _populate_amphora_api_version /usr/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py:112
2022-04-08 13:45:16.664 38 DEBUG octavia.amphorae.drivers.haproxy.rest_api_driver [req-627fe0bb-cbb1-491e-a682-ccf199cb577f - 2886febc2f0c44fea2250ec811834f37 - - -] Amphora 3d8868f5-088f-44e7-88b4-fe860f2f0972 for loadbalancer 482777a1-269c-4872-9a36-b883f08c1902 is already in single process mode. update_amphora_listeners /usr/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py:150
2022-04-08 13:45:16.664 38 DEBUG octavia.amphorae.drivers.haproxy.rest_api_driver [req-627fe0bb-cbb1-491e-a682-ccf199cb577f - 2886febc2f0c44fea2250ec811834f37 - - -] HaproxyAmphoraLoadBalancerDriver updating listener 70ebb045-83de-47bc-ac39-46fd86c29f45 on amphora 3d8868f5-088f-44e7-88b4-fe860f2f0972 update_amphora_listeners /usr/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py:157
2022-04-08 13:45:16.665 38 DEBUG octavia.amphorae.drivers.haproxy.rest_api_driver [req-4ed5f5fa-ed1b-44b0-a48b-4868c2e7f52a - b770a0d5d13744fface4b7406fbc4805 - - -] request url loadbalancer/3d8868f5-088f-44e7-88b4-fe860f2f0972/482777a1-269c-4872-9a36-b883f08c1902/haproxy request /usr/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py:678
2022-04-08 13:45:16.665 38 DEBUG octavia.amphorae.drivers.haproxy.rest_api_driver [req-4ed5f5fa-ed1b-44b0-a48b-4868c2e7f52a - b770a0d5d13744fface4b7406fbc4805 - - -] request url https://172.24.3.163:9443/1.0/loadbalancer/3d8868f5-088f-44e7-88b4-fe860f2f0972/482777a1-269c-4872-9a36-b883f08c1902/haproxy request /usr/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py:681
2022-04-08 13:45:17.302 38 DEBUG octavia.amphorae.drivers.haproxy.rest_api_driver [req-4ed5f5fa-ed1b-44b0-a48b-4868c2e7f52a - b770a0d5d13744fface4b7406fbc4805 - - -] Connected to amphora. Response: <Response [202]> request /usr/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py:702
2022-04-08 13:45:17.303 38 DEBUG octavia.amphorae.drivers.haproxy.rest_api_driver [req-4ed5f5fa-ed1b-44b0-a48b-4868c2e7f52a - b770a0d5d13744fface4b7406fbc4805 - - -] request url loadbalancer/482777a1-269c-4872-9a36-b883f08c1902/reload request /usr/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py:678
2022-04-08 13:45:17.303 38 DEBUG octavia.amphorae.drivers.haproxy.rest_api_driver [req-4ed5f5fa-ed1b-44b0-a48b-4868c2e7f52a - b770a0d5d13744fface4b7406fbc4805 - - -] request url https://172.24.3.163:9443/1.0/loadbalancer/482777a1-269c-4872-9a36-b883f08c1902/reload request /usr/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py:681
2022-04-08 13:45:17.382 38 DEBUG octavia.amphorae.drivers.haproxy.rest_api_driver [req-4ed5f5fa-ed1b-44b0-a48b-4868c2e7f52a - b770a0d5d13744fface4b7406fbc4805 - - -] Connected to amphora. Response: <Response [500]> request /usr/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py:702
2022-04-08 13:45:17.382 38 ERROR octavia.amphorae.drivers.haproxy.exceptions [req-4ed5f5fa-ed1b-44b0-a48b-4868c2e7f52a - b770a0d5d13744fface4b7406fbc4805 - - -] Amphora agent returned unexpected result code 500 with response {'message': 'Error reloading haproxy', 'details': 'Redirecting to /bin/systemctl reload haproxy-482777a1-269c-4872-9a36-b883f08c1902.service\nJob for haproxy-482777a1-269c-4872-9a36-b883f08c1902.service failed.\nSee "systemctl status haproxy-482777a1-269c-4872-9a36-b883f08c1902.service" and "journalctl -xe" for details.\n'}
2022-04-08 13:45:17.385 38 WARNING octavia.controller.worker.v1.controller_worker [-] Task 'octavia.controller.worker.v1.tasks.amphora_driver_tasks.ListenersUpdate' (66ff05b8-0756-4e4d-85df-f09a71805b4b) transitioned into state 'FAILURE' from state 'RUNNING'


In the amphora logs:

Apr 08 09:45:17 amphora-3d8868f5-088f-44e7-88b4-fe860f2f0972 systemd[1]: Reloading HAProxy Load Balancer.
Apr 08 09:45:17 amphora-3d8868f5-088f-44e7-88b4-fe860f2f0972 systemd[5808]: haproxy-482777a1-269c-4872-9a36-b883f08c1902.service: Failed to execute command: Permission denied
Apr 08 09:45:17 amphora-3d8868f5-088f-44e7-88b4-fe860f2f0972 systemd[5808]: haproxy-482777a1-269c-4872-9a36-b883f08c1902.service: Failed at step EXEC spawning /bin/sh: Permission denied
Apr 08 09:45:17 amphora-3d8868f5-088f-44e7-88b4-fe860f2f0972 systemd[1]: haproxy-482777a1-269c-4872-9a36-b883f08c1902.service: Control process exited, code=exited status=203
Apr 08 09:45:17 amphora-3d8868f5-088f-44e7-88b4-fe860f2f0972 amphora-agent[1425]: 2022-04-08 09:45:17.126 1425 DEBUG octavia.amphorae.backends.agent.api_server.loadbalancer [-] Failed to reload haproxy-482777a1-269c-4872-9a36-b883f08c1902 service: Command '['/usr/sbin/service', 'haproxy-482777a1-269c-4872-9a36-b883f08c1902', 'reload']' returned non-zero exit status 1. b'Redirecting to /bin/systemctl reload haproxy-482777a1-269c-4872-9a36-b883f08c1902.service\nJob for haproxy-482777a1-269c-4872-9a36-b883f08c1902.service failed.\nSee "systemctl status haproxy-482777a1-269c-4872-9a36-b883f08c1902.service" and "journalctl -xe" for details.\n' start_stop_lb /usr/lib/python3.6/site-packages/octavia/amphorae/backends/agent/api_server/loadbalancer.py:258
Apr 08 09:45:17 amphora-3d8868f5-088f-44e7-88b4-fe860f2f0972 amphora-agent[1066]: 2022-04-08 09:45:17.126 1425 DEBUG octavia.amphorae.backends.agent.api_server.loadbalancer [-] Failed to reload haproxy-482777a1-269c-4872-9a36-b883f08c1902 service: Command '['/usr/sbin/service', 'haproxy-482777a1-269c-4872-9a36-b883f08c1902', 'reload']' returned non-zero exit status 1. b'Redirecting to /bin/systemctl reload haproxy-482777a1-269c-4872-9a36-b883f08c1902.service\nJob for haproxy-482777a1-269c-4872-9a36-b883f08c1902.service failed.\nSee "systemctl status haproxy-482777a1-269c-4872-9a36-b883f08c1902.service" and "journalctl -xe" for details.\n' start_stop_lb /usr/lib/python3.6/site-packages/octavia/amphorae/backends/agent/api_server/loadbalancer.py:258
Apr 08 09:45:17 amphora-3d8868f5-088f-44e7-88b4-fe860f2f0972 systemd[1]: Reload failed for HAProxy Load Balancer.

/var/log/audit/audit.log in the amp:

type=SERVICE_START msg=audit(1649425399.455:193): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=haproxy-482777a1-269c-4872-9a36-b883f08c1902 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'UID="root" AUID="unset"
type=AVC msg=audit(1649425422.841:194): avc:  denied  { entrypoint } for  pid=5633 comm="(sh)" path="/usr/bin/bash" dev="vda1" ino=4215617 scontext=system_u:system_r:haproxy_t:s0 tcontext=system_u:object_r:shell_exec_t:s0 tclass=file permissive=0


Version-Release number of selected component (if applicable):
17.0

How reproducible:
100%

Steps to Reproduce:
1. Create a LB, a listener, then create a pool, the amphora returns an error

Comment 12 Omer Schwartz 2022-05-12 14:39:03 UTC

Verified on puddle RHOS-16.2-RHEL-8-20220427.n.3

(overcloud) [stack@undercloud-0 ~]$ cat /etc/rhosp-release 
Red Hat OpenStack Platform release 16.2.2 (Train)

# Creating the LB
(overcloud) [stack@undercloud-0 ~]$ openstack loadbalancer create --vip-subnet-id int_sub --enable --name BZ2078539_lb
+---------------------+--------------------------------------+
| Field               | Value                                |
+---------------------+--------------------------------------+
| admin_state_up      | True                                 |
| created_at          | 2022-05-12T14:26:50                  |
| description         |                                      |
| flavor_id           | None                                 |
| id                  | acc4e9d7-52ba-48a8-ae49-4dd0b6551de2 |
| listeners           |                                      |
| name                | BZ2078539_lb                         |
| operating_status    | OFFLINE                              |
| pools               |                                      |
| project_id          | bf4831c9da594f0cb8935ea1f8f2bf75     |
| provider            | amphora                              |
| provisioning_status | PENDING_CREATE                       |
| updated_at          | None                                 |
| vip_address         | 192.168.1.224                        |
| vip_network_id      | 322efa48-bfb6-416f-929b-81331ba33d7e |
| vip_port_id         | ad0c00f3-79a4-444f-b7ed-d7e37ec8c429 |
| vip_qos_policy_id   | None                                 |
| vip_subnet_id       | 507742f2-990c-488c-9c1f-19f766687925 |
+---------------------+--------------------------------------+

# Creating the pool
(overcloud) [stack@undercloud-0 ~]$ openstack loadbalancer pool create --protocol HTTP --loadbalancer acc4e9d7-52ba-48a8-ae49-4dd0b6551de2 --lb-algorithm ROUND_ROBIN
+----------------------+--------------------------------------+
| Field                | Value                                |
+----------------------+--------------------------------------+
| admin_state_up       | True                                 |
| created_at           | 2022-05-12T14:35:54                  |
| description          |                                      |
| healthmonitor_id     |                                      |
| id                   | 5b489459-1605-4082-95c2-71c359a15b3b |
| lb_algorithm         | ROUND_ROBIN                          |
| listeners            |                                      |
| loadbalancers        | acc4e9d7-52ba-48a8-ae49-4dd0b6551de2 |
| members              |                                      |
| name                 |                                      |
| operating_status     | OFFLINE                              |
| project_id           | bf4831c9da594f0cb8935ea1f8f2bf75     |
| protocol             | HTTP                                 |
| provisioning_status  | PENDING_CREATE                       |
| session_persistence  | None                                 |
| updated_at           | None                                 |
| tls_container_ref    | None                                 |
| ca_tls_container_ref | None                                 |
| crl_container_ref    | None                                 |
| tls_enabled          | False                                |
+----------------------+--------------------------------------+




# Verifying both the LB and the amphora provisioning_status/status are ACTIVE/ALLOCATED:

(overcloud) [stack@undercloud-0 ~]$ openstack loadbalancer list | grep BZ
| acc4e9d7-52ba-48a8-ae49-4dd0b6551de2 | BZ2078539_lb                                                                            | bf4831c9da594f0cb8935ea1f8f2bf75 | 192.168.1.224 | ACTIVE              | amphora  |


(overcloud) [stack@undercloud-0 ~]$ openstack loadbalancer amphora list | grep acc4e9d7-52ba-48a8-ae49-4dd0b6551de2
| 84ca3ff9-77e4-4e8d-a615-212109b665fc | acc4e9d7-52ba-48a8-ae49-4dd0b6551de2 | ALLOCATED | BACKUP | 172.24.3.240  | 192.168.1.224 |
| aa22681e-b8cc-4d33-996c-1eeb2b326741 | acc4e9d7-52ba-48a8-ae49-4dd0b6551de2 | ALLOCATED | MASTER | 172.24.3.214  | 192.168.1.224 |

Looks good to me, verified.

Comment 17 errata-xmlrpc 2022-06-22 16:06:30 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Release of components for Red Hat OpenStack Platform 16.2.3 (Train)), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:4793

Note You need to log in before you can comment on or make changes to this bug.