Bug 2078539

Summary: Octavia cannot reload haproxy because of selinux policies
Product: Red Hat OpenStack Reporter: Gregory Thiemonge <gthiemon>
Component: openstack-octaviaAssignee: Gregory Thiemonge <gthiemon>
Status: CLOSED ERRATA QA Contact: Bruna Bonguardo <bbonguar>
Severity: high Docs Contact:
Priority: high    
Version: 16.2 (Train)CC: bbonguar, cjeanner, ihrachys, jpichon, lhh, lpeer, lvrabec, majopela, njohnston, oschwart, scohen, slinaber, spower, wznoinsk
Target Milestone: z3Keywords: AutomationBlocker, Triaged
Target Release: 16.2 (Train on RHEL 8.4)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-octavia-5.1.3-2.20220328185156.el8ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 2073491
: 2136558 (view as bug list) Environment:
Last Closed: 2022-06-22 16:06:30 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 2073491    
Bug Blocks: 2136558    

Description Gregory Thiemonge 2022-04-25 14:31:42 UTC
+++ This bug was initially created as a clone of Bug #2073491 +++

Description of problem:

In OSP17 (RHEL8 and 9), Octavia fails to reload haproxy after each configuration update.

The worker logs show:

2022-04-08 13:45:16.578 38 DEBUG octavia.controller.worker.v1.controller_worker [-] Task 'octavia.controller.worker.v1.tasks.amphora_driver_tasks.ListenersUpdate' (66ff05b8-0756-4e4d-85df-f09a71805b4b) transitioned into state 'RUNNING' from state 'PENDING' _task_receiver /usr/lib/python3.6/site-packages/taskflow/listeners/logging.py:192
2022-04-08 13:45:16.578 38 DEBUG octavia.amphorae.drivers.haproxy.rest_api_driver [req-627fe0bb-cbb1-491e-a682-ccf199cb577f - 2886febc2f0c44fea2250ec811834f37 - - -] request url / request /usr/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py:678
2022-04-08 13:45:16.579 38 DEBUG octavia.amphorae.drivers.haproxy.rest_api_driver [req-627fe0bb-cbb1-491e-a682-ccf199cb577f - 2886febc2f0c44fea2250ec811834f37 - - -] request url https://172.24.3.163:9443// request /usr/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py:681
2022-04-08 13:45:16.597 38 DEBUG octavia.amphorae.drivers.haproxy.rest_api_driver [req-627fe0bb-cbb1-491e-a682-ccf199cb577f - 2886febc2f0c44fea2250ec811834f37 - - -] Connected to amphora. Response: <Response [200]> request /usr/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py:702
2022-04-08 13:45:16.597 38 DEBUG octavia.amphorae.drivers.haproxy.rest_api_driver [req-627fe0bb-cbb1-491e-a682-ccf199cb577f - 2886febc2f0c44fea2250ec811834f37 - - -] Amphora 3d8868f5-088f-44e7-88b4-fe860f2f0972 has API version 1.0 _populate_amphora_api_version /usr/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py:112
2022-04-08 13:45:16.598 38 DEBUG octavia.amphorae.drivers.haproxy.rest_api_driver [req-627fe0bb-cbb1-491e-a682-ccf199cb577f - 2886febc2f0c44fea2250ec811834f37 - - -] request url info request /usr/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py:678
2022-04-08 13:45:16.598 38 DEBUG octavia.amphorae.drivers.haproxy.rest_api_driver [req-627fe0bb-cbb1-491e-a682-ccf199cb577f - 2886febc2f0c44fea2250ec811834f37 - - -] request url https://172.24.3.163:9443/1.0/info request /usr/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py:681
2022-04-08 13:45:16.663 38 DEBUG octavia.amphorae.drivers.haproxy.rest_api_driver [req-627fe0bb-cbb1-491e-a682-ccf199cb577f - 2886febc2f0c44fea2250ec811834f37 - - -] Connected to amphora. Response: <Response [200]> request /usr/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py:702
2022-04-08 13:45:16.663 38 DEBUG octavia.amphorae.drivers.haproxy.rest_api_driver [req-627fe0bb-cbb1-491e-a682-ccf199cb577f - 2886febc2f0c44fea2250ec811834f37 - - -] Amphora 3d8868f5-088f-44e7-88b4-fe860f2f0972 has API version 1.0 _populate_amphora_api_version /usr/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py:112
2022-04-08 13:45:16.664 38 DEBUG octavia.amphorae.drivers.haproxy.rest_api_driver [req-627fe0bb-cbb1-491e-a682-ccf199cb577f - 2886febc2f0c44fea2250ec811834f37 - - -] Amphora 3d8868f5-088f-44e7-88b4-fe860f2f0972 for loadbalancer 482777a1-269c-4872-9a36-b883f08c1902 is already in single process mode. update_amphora_listeners /usr/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py:150
2022-04-08 13:45:16.664 38 DEBUG octavia.amphorae.drivers.haproxy.rest_api_driver [req-627fe0bb-cbb1-491e-a682-ccf199cb577f - 2886febc2f0c44fea2250ec811834f37 - - -] HaproxyAmphoraLoadBalancerDriver updating listener 70ebb045-83de-47bc-ac39-46fd86c29f45 on amphora 3d8868f5-088f-44e7-88b4-fe860f2f0972 update_amphora_listeners /usr/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py:157
2022-04-08 13:45:16.665 38 DEBUG octavia.amphorae.drivers.haproxy.rest_api_driver [req-4ed5f5fa-ed1b-44b0-a48b-4868c2e7f52a - b770a0d5d13744fface4b7406fbc4805 - - -] request url loadbalancer/3d8868f5-088f-44e7-88b4-fe860f2f0972/482777a1-269c-4872-9a36-b883f08c1902/haproxy request /usr/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py:678
2022-04-08 13:45:16.665 38 DEBUG octavia.amphorae.drivers.haproxy.rest_api_driver [req-4ed5f5fa-ed1b-44b0-a48b-4868c2e7f52a - b770a0d5d13744fface4b7406fbc4805 - - -] request url https://172.24.3.163:9443/1.0/loadbalancer/3d8868f5-088f-44e7-88b4-fe860f2f0972/482777a1-269c-4872-9a36-b883f08c1902/haproxy request /usr/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py:681
2022-04-08 13:45:17.302 38 DEBUG octavia.amphorae.drivers.haproxy.rest_api_driver [req-4ed5f5fa-ed1b-44b0-a48b-4868c2e7f52a - b770a0d5d13744fface4b7406fbc4805 - - -] Connected to amphora. Response: <Response [202]> request /usr/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py:702
2022-04-08 13:45:17.303 38 DEBUG octavia.amphorae.drivers.haproxy.rest_api_driver [req-4ed5f5fa-ed1b-44b0-a48b-4868c2e7f52a - b770a0d5d13744fface4b7406fbc4805 - - -] request url loadbalancer/482777a1-269c-4872-9a36-b883f08c1902/reload request /usr/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py:678
2022-04-08 13:45:17.303 38 DEBUG octavia.amphorae.drivers.haproxy.rest_api_driver [req-4ed5f5fa-ed1b-44b0-a48b-4868c2e7f52a - b770a0d5d13744fface4b7406fbc4805 - - -] request url https://172.24.3.163:9443/1.0/loadbalancer/482777a1-269c-4872-9a36-b883f08c1902/reload request /usr/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py:681
2022-04-08 13:45:17.382 38 DEBUG octavia.amphorae.drivers.haproxy.rest_api_driver [req-4ed5f5fa-ed1b-44b0-a48b-4868c2e7f52a - b770a0d5d13744fface4b7406fbc4805 - - -] Connected to amphora. Response: <Response [500]> request /usr/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py:702
2022-04-08 13:45:17.382 38 ERROR octavia.amphorae.drivers.haproxy.exceptions [req-4ed5f5fa-ed1b-44b0-a48b-4868c2e7f52a - b770a0d5d13744fface4b7406fbc4805 - - -] Amphora agent returned unexpected result code 500 with response {'message': 'Error reloading haproxy', 'details': 'Redirecting to /bin/systemctl reload haproxy-482777a1-269c-4872-9a36-b883f08c1902.service\nJob for haproxy-482777a1-269c-4872-9a36-b883f08c1902.service failed.\nSee "systemctl status haproxy-482777a1-269c-4872-9a36-b883f08c1902.service" and "journalctl -xe" for details.\n'}
2022-04-08 13:45:17.385 38 WARNING octavia.controller.worker.v1.controller_worker [-] Task 'octavia.controller.worker.v1.tasks.amphora_driver_tasks.ListenersUpdate' (66ff05b8-0756-4e4d-85df-f09a71805b4b) transitioned into state 'FAILURE' from state 'RUNNING'


In the amphora logs:

Apr 08 09:45:17 amphora-3d8868f5-088f-44e7-88b4-fe860f2f0972 systemd[1]: Reloading HAProxy Load Balancer.
Apr 08 09:45:17 amphora-3d8868f5-088f-44e7-88b4-fe860f2f0972 systemd[5808]: haproxy-482777a1-269c-4872-9a36-b883f08c1902.service: Failed to execute command: Permission denied
Apr 08 09:45:17 amphora-3d8868f5-088f-44e7-88b4-fe860f2f0972 systemd[5808]: haproxy-482777a1-269c-4872-9a36-b883f08c1902.service: Failed at step EXEC spawning /bin/sh: Permission denied
Apr 08 09:45:17 amphora-3d8868f5-088f-44e7-88b4-fe860f2f0972 systemd[1]: haproxy-482777a1-269c-4872-9a36-b883f08c1902.service: Control process exited, code=exited status=203
Apr 08 09:45:17 amphora-3d8868f5-088f-44e7-88b4-fe860f2f0972 amphora-agent[1425]: 2022-04-08 09:45:17.126 1425 DEBUG octavia.amphorae.backends.agent.api_server.loadbalancer [-] Failed to reload haproxy-482777a1-269c-4872-9a36-b883f08c1902 service: Command '['/usr/sbin/service', 'haproxy-482777a1-269c-4872-9a36-b883f08c1902', 'reload']' returned non-zero exit status 1. b'Redirecting to /bin/systemctl reload haproxy-482777a1-269c-4872-9a36-b883f08c1902.service\nJob for haproxy-482777a1-269c-4872-9a36-b883f08c1902.service failed.\nSee "systemctl status haproxy-482777a1-269c-4872-9a36-b883f08c1902.service" and "journalctl -xe" for details.\n' start_stop_lb /usr/lib/python3.6/site-packages/octavia/amphorae/backends/agent/api_server/loadbalancer.py:258
Apr 08 09:45:17 amphora-3d8868f5-088f-44e7-88b4-fe860f2f0972 amphora-agent[1066]: 2022-04-08 09:45:17.126 1425 DEBUG octavia.amphorae.backends.agent.api_server.loadbalancer [-] Failed to reload haproxy-482777a1-269c-4872-9a36-b883f08c1902 service: Command '['/usr/sbin/service', 'haproxy-482777a1-269c-4872-9a36-b883f08c1902', 'reload']' returned non-zero exit status 1. b'Redirecting to /bin/systemctl reload haproxy-482777a1-269c-4872-9a36-b883f08c1902.service\nJob for haproxy-482777a1-269c-4872-9a36-b883f08c1902.service failed.\nSee "systemctl status haproxy-482777a1-269c-4872-9a36-b883f08c1902.service" and "journalctl -xe" for details.\n' start_stop_lb /usr/lib/python3.6/site-packages/octavia/amphorae/backends/agent/api_server/loadbalancer.py:258
Apr 08 09:45:17 amphora-3d8868f5-088f-44e7-88b4-fe860f2f0972 systemd[1]: Reload failed for HAProxy Load Balancer.

/var/log/audit/audit.log in the amp:

type=SERVICE_START msg=audit(1649425399.455:193): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=haproxy-482777a1-269c-4872-9a36-b883f08c1902 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'UID="root" AUID="unset"
type=AVC msg=audit(1649425422.841:194): avc:  denied  { entrypoint } for  pid=5633 comm="(sh)" path="/usr/bin/bash" dev="vda1" ino=4215617 scontext=system_u:system_r:haproxy_t:s0 tcontext=system_u:object_r:shell_exec_t:s0 tclass=file permissive=0


Version-Release number of selected component (if applicable):
17.0

How reproducible:
100%

Steps to Reproduce:
1. Create a LB, a listener, then create a pool, the amphora returns an error

Comment 12 Omer Schwartz 2022-05-12 14:39:03 UTC
Verified on puddle RHOS-16.2-RHEL-8-20220427.n.3

(overcloud) [stack@undercloud-0 ~]$ cat /etc/rhosp-release 
Red Hat OpenStack Platform release 16.2.2 (Train)

# Creating the LB
(overcloud) [stack@undercloud-0 ~]$ openstack loadbalancer create --vip-subnet-id int_sub --enable --name BZ2078539_lb
+---------------------+--------------------------------------+
| Field               | Value                                |
+---------------------+--------------------------------------+
| admin_state_up      | True                                 |
| created_at          | 2022-05-12T14:26:50                  |
| description         |                                      |
| flavor_id           | None                                 |
| id                  | acc4e9d7-52ba-48a8-ae49-4dd0b6551de2 |
| listeners           |                                      |
| name                | BZ2078539_lb                         |
| operating_status    | OFFLINE                              |
| pools               |                                      |
| project_id          | bf4831c9da594f0cb8935ea1f8f2bf75     |
| provider            | amphora                              |
| provisioning_status | PENDING_CREATE                       |
| updated_at          | None                                 |
| vip_address         | 192.168.1.224                        |
| vip_network_id      | 322efa48-bfb6-416f-929b-81331ba33d7e |
| vip_port_id         | ad0c00f3-79a4-444f-b7ed-d7e37ec8c429 |
| vip_qos_policy_id   | None                                 |
| vip_subnet_id       | 507742f2-990c-488c-9c1f-19f766687925 |
+---------------------+--------------------------------------+

# Creating the pool
(overcloud) [stack@undercloud-0 ~]$ openstack loadbalancer pool create --protocol HTTP --loadbalancer acc4e9d7-52ba-48a8-ae49-4dd0b6551de2 --lb-algorithm ROUND_ROBIN
+----------------------+--------------------------------------+
| Field                | Value                                |
+----------------------+--------------------------------------+
| admin_state_up       | True                                 |
| created_at           | 2022-05-12T14:35:54                  |
| description          |                                      |
| healthmonitor_id     |                                      |
| id                   | 5b489459-1605-4082-95c2-71c359a15b3b |
| lb_algorithm         | ROUND_ROBIN                          |
| listeners            |                                      |
| loadbalancers        | acc4e9d7-52ba-48a8-ae49-4dd0b6551de2 |
| members              |                                      |
| name                 |                                      |
| operating_status     | OFFLINE                              |
| project_id           | bf4831c9da594f0cb8935ea1f8f2bf75     |
| protocol             | HTTP                                 |
| provisioning_status  | PENDING_CREATE                       |
| session_persistence  | None                                 |
| updated_at           | None                                 |
| tls_container_ref    | None                                 |
| ca_tls_container_ref | None                                 |
| crl_container_ref    | None                                 |
| tls_enabled          | False                                |
+----------------------+--------------------------------------+




# Verifying both the LB and the amphora provisioning_status/status are ACTIVE/ALLOCATED:

(overcloud) [stack@undercloud-0 ~]$ openstack loadbalancer list | grep BZ
| acc4e9d7-52ba-48a8-ae49-4dd0b6551de2 | BZ2078539_lb                                                                            | bf4831c9da594f0cb8935ea1f8f2bf75 | 192.168.1.224 | ACTIVE              | amphora  |


(overcloud) [stack@undercloud-0 ~]$ openstack loadbalancer amphora list | grep acc4e9d7-52ba-48a8-ae49-4dd0b6551de2
| 84ca3ff9-77e4-4e8d-a615-212109b665fc | acc4e9d7-52ba-48a8-ae49-4dd0b6551de2 | ALLOCATED | BACKUP | 172.24.3.240  | 192.168.1.224 |
| aa22681e-b8cc-4d33-996c-1eeb2b326741 | acc4e9d7-52ba-48a8-ae49-4dd0b6551de2 | ALLOCATED | MASTER | 172.24.3.214  | 192.168.1.224 |

Looks good to me, verified.

Comment 17 errata-xmlrpc 2022-06-22 16:06:30 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Release of components for Red Hat OpenStack Platform 16.2.3 (Train)), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:4793