Bug 2073491

Summary: Octavia cannot reload haproxy because of selinux policies
Product: Red Hat OpenStack Reporter: Gregory Thiemonge <gthiemon>
Component: openstack-octaviaAssignee: Gregory Thiemonge <gthiemon>
Status: CLOSED ERRATA QA Contact: Bruna Bonguardo <bbonguar>
Severity: high Docs Contact:
Priority: high    
Version: 17.0 (Wallaby)CC: cjeanner, ihrachys, jpichon, lhh, lpeer, lvrabec, majopela, oschwart, scohen, wznoinsk
Target Milestone: AlphaKeywords: AutomationBlocker, Triaged
Target Release: 17.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-octavia-8.0.2-0.20220422120539.1329b57.el9ost Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of:
: 2078539 (view as bug list) Environment:
Last Closed: 2022-09-21 12:20:36 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 2078539, 2136558    

Description Gregory Thiemonge 2022-04-08 15:33:41 UTC
Description of problem:

In OSP17 (RHEL8 and 9), Octavia fails to reload haproxy after each configuration update.

The worker logs show:

2022-04-08 13:45:16.578 38 DEBUG octavia.controller.worker.v1.controller_worker [-] Task 'octavia.controller.worker.v1.tasks.amphora_driver_tasks.ListenersUpdate' (66ff05b8-0756-4e4d-85df-f09a71805b4b) transitioned into state 'RUNNING' from state 'PENDING' _task_receiver /usr/lib/python3.6/site-packages/taskflow/listeners/logging.py:192
2022-04-08 13:45:16.578 38 DEBUG octavia.amphorae.drivers.haproxy.rest_api_driver [req-627fe0bb-cbb1-491e-a682-ccf199cb577f - 2886febc2f0c44fea2250ec811834f37 - - -] request url / request /usr/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py:678
2022-04-08 13:45:16.579 38 DEBUG octavia.amphorae.drivers.haproxy.rest_api_driver [req-627fe0bb-cbb1-491e-a682-ccf199cb577f - 2886febc2f0c44fea2250ec811834f37 - - -] request url https://172.24.3.163:9443// request /usr/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py:681
2022-04-08 13:45:16.597 38 DEBUG octavia.amphorae.drivers.haproxy.rest_api_driver [req-627fe0bb-cbb1-491e-a682-ccf199cb577f - 2886febc2f0c44fea2250ec811834f37 - - -] Connected to amphora. Response: <Response [200]> request /usr/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py:702
2022-04-08 13:45:16.597 38 DEBUG octavia.amphorae.drivers.haproxy.rest_api_driver [req-627fe0bb-cbb1-491e-a682-ccf199cb577f - 2886febc2f0c44fea2250ec811834f37 - - -] Amphora 3d8868f5-088f-44e7-88b4-fe860f2f0972 has API version 1.0 _populate_amphora_api_version /usr/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py:112
2022-04-08 13:45:16.598 38 DEBUG octavia.amphorae.drivers.haproxy.rest_api_driver [req-627fe0bb-cbb1-491e-a682-ccf199cb577f - 2886febc2f0c44fea2250ec811834f37 - - -] request url info request /usr/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py:678
2022-04-08 13:45:16.598 38 DEBUG octavia.amphorae.drivers.haproxy.rest_api_driver [req-627fe0bb-cbb1-491e-a682-ccf199cb577f - 2886febc2f0c44fea2250ec811834f37 - - -] request url https://172.24.3.163:9443/1.0/info request /usr/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py:681
2022-04-08 13:45:16.663 38 DEBUG octavia.amphorae.drivers.haproxy.rest_api_driver [req-627fe0bb-cbb1-491e-a682-ccf199cb577f - 2886febc2f0c44fea2250ec811834f37 - - -] Connected to amphora. Response: <Response [200]> request /usr/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py:702
2022-04-08 13:45:16.663 38 DEBUG octavia.amphorae.drivers.haproxy.rest_api_driver [req-627fe0bb-cbb1-491e-a682-ccf199cb577f - 2886febc2f0c44fea2250ec811834f37 - - -] Amphora 3d8868f5-088f-44e7-88b4-fe860f2f0972 has API version 1.0 _populate_amphora_api_version /usr/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py:112
2022-04-08 13:45:16.664 38 DEBUG octavia.amphorae.drivers.haproxy.rest_api_driver [req-627fe0bb-cbb1-491e-a682-ccf199cb577f - 2886febc2f0c44fea2250ec811834f37 - - -] Amphora 3d8868f5-088f-44e7-88b4-fe860f2f0972 for loadbalancer 482777a1-269c-4872-9a36-b883f08c1902 is already in single process mode. update_amphora_listeners /usr/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py:150
2022-04-08 13:45:16.664 38 DEBUG octavia.amphorae.drivers.haproxy.rest_api_driver [req-627fe0bb-cbb1-491e-a682-ccf199cb577f - 2886febc2f0c44fea2250ec811834f37 - - -] HaproxyAmphoraLoadBalancerDriver updating listener 70ebb045-83de-47bc-ac39-46fd86c29f45 on amphora 3d8868f5-088f-44e7-88b4-fe860f2f0972 update_amphora_listeners /usr/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py:157
2022-04-08 13:45:16.665 38 DEBUG octavia.amphorae.drivers.haproxy.rest_api_driver [req-4ed5f5fa-ed1b-44b0-a48b-4868c2e7f52a - b770a0d5d13744fface4b7406fbc4805 - - -] request url loadbalancer/3d8868f5-088f-44e7-88b4-fe860f2f0972/482777a1-269c-4872-9a36-b883f08c1902/haproxy request /usr/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py:678
2022-04-08 13:45:16.665 38 DEBUG octavia.amphorae.drivers.haproxy.rest_api_driver [req-4ed5f5fa-ed1b-44b0-a48b-4868c2e7f52a - b770a0d5d13744fface4b7406fbc4805 - - -] request url https://172.24.3.163:9443/1.0/loadbalancer/3d8868f5-088f-44e7-88b4-fe860f2f0972/482777a1-269c-4872-9a36-b883f08c1902/haproxy request /usr/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py:681
2022-04-08 13:45:17.302 38 DEBUG octavia.amphorae.drivers.haproxy.rest_api_driver [req-4ed5f5fa-ed1b-44b0-a48b-4868c2e7f52a - b770a0d5d13744fface4b7406fbc4805 - - -] Connected to amphora. Response: <Response [202]> request /usr/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py:702
2022-04-08 13:45:17.303 38 DEBUG octavia.amphorae.drivers.haproxy.rest_api_driver [req-4ed5f5fa-ed1b-44b0-a48b-4868c2e7f52a - b770a0d5d13744fface4b7406fbc4805 - - -] request url loadbalancer/482777a1-269c-4872-9a36-b883f08c1902/reload request /usr/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py:678
2022-04-08 13:45:17.303 38 DEBUG octavia.amphorae.drivers.haproxy.rest_api_driver [req-4ed5f5fa-ed1b-44b0-a48b-4868c2e7f52a - b770a0d5d13744fface4b7406fbc4805 - - -] request url https://172.24.3.163:9443/1.0/loadbalancer/482777a1-269c-4872-9a36-b883f08c1902/reload request /usr/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py:681
2022-04-08 13:45:17.382 38 DEBUG octavia.amphorae.drivers.haproxy.rest_api_driver [req-4ed5f5fa-ed1b-44b0-a48b-4868c2e7f52a - b770a0d5d13744fface4b7406fbc4805 - - -] Connected to amphora. Response: <Response [500]> request /usr/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py:702
2022-04-08 13:45:17.382 38 ERROR octavia.amphorae.drivers.haproxy.exceptions [req-4ed5f5fa-ed1b-44b0-a48b-4868c2e7f52a - b770a0d5d13744fface4b7406fbc4805 - - -] Amphora agent returned unexpected result code 500 with response {'message': 'Error reloading haproxy', 'details': 'Redirecting to /bin/systemctl reload haproxy-482777a1-269c-4872-9a36-b883f08c1902.service\nJob for haproxy-482777a1-269c-4872-9a36-b883f08c1902.service failed.\nSee "systemctl status haproxy-482777a1-269c-4872-9a36-b883f08c1902.service" and "journalctl -xe" for details.\n'}
2022-04-08 13:45:17.385 38 WARNING octavia.controller.worker.v1.controller_worker [-] Task 'octavia.controller.worker.v1.tasks.amphora_driver_tasks.ListenersUpdate' (66ff05b8-0756-4e4d-85df-f09a71805b4b) transitioned into state 'FAILURE' from state 'RUNNING'


In the amphora logs:

Apr 08 09:45:17 amphora-3d8868f5-088f-44e7-88b4-fe860f2f0972 systemd[1]: Reloading HAProxy Load Balancer.
Apr 08 09:45:17 amphora-3d8868f5-088f-44e7-88b4-fe860f2f0972 systemd[5808]: haproxy-482777a1-269c-4872-9a36-b883f08c1902.service: Failed to execute command: Permission denied
Apr 08 09:45:17 amphora-3d8868f5-088f-44e7-88b4-fe860f2f0972 systemd[5808]: haproxy-482777a1-269c-4872-9a36-b883f08c1902.service: Failed at step EXEC spawning /bin/sh: Permission denied
Apr 08 09:45:17 amphora-3d8868f5-088f-44e7-88b4-fe860f2f0972 systemd[1]: haproxy-482777a1-269c-4872-9a36-b883f08c1902.service: Control process exited, code=exited status=203
Apr 08 09:45:17 amphora-3d8868f5-088f-44e7-88b4-fe860f2f0972 amphora-agent[1425]: 2022-04-08 09:45:17.126 1425 DEBUG octavia.amphorae.backends.agent.api_server.loadbalancer [-] Failed to reload haproxy-482777a1-269c-4872-9a36-b883f08c1902 service: Command '['/usr/sbin/service', 'haproxy-482777a1-269c-4872-9a36-b883f08c1902', 'reload']' returned non-zero exit status 1. b'Redirecting to /bin/systemctl reload haproxy-482777a1-269c-4872-9a36-b883f08c1902.service\nJob for haproxy-482777a1-269c-4872-9a36-b883f08c1902.service failed.\nSee "systemctl status haproxy-482777a1-269c-4872-9a36-b883f08c1902.service" and "journalctl -xe" for details.\n' start_stop_lb /usr/lib/python3.6/site-packages/octavia/amphorae/backends/agent/api_server/loadbalancer.py:258
Apr 08 09:45:17 amphora-3d8868f5-088f-44e7-88b4-fe860f2f0972 amphora-agent[1066]: 2022-04-08 09:45:17.126 1425 DEBUG octavia.amphorae.backends.agent.api_server.loadbalancer [-] Failed to reload haproxy-482777a1-269c-4872-9a36-b883f08c1902 service: Command '['/usr/sbin/service', 'haproxy-482777a1-269c-4872-9a36-b883f08c1902', 'reload']' returned non-zero exit status 1. b'Redirecting to /bin/systemctl reload haproxy-482777a1-269c-4872-9a36-b883f08c1902.service\nJob for haproxy-482777a1-269c-4872-9a36-b883f08c1902.service failed.\nSee "systemctl status haproxy-482777a1-269c-4872-9a36-b883f08c1902.service" and "journalctl -xe" for details.\n' start_stop_lb /usr/lib/python3.6/site-packages/octavia/amphorae/backends/agent/api_server/loadbalancer.py:258
Apr 08 09:45:17 amphora-3d8868f5-088f-44e7-88b4-fe860f2f0972 systemd[1]: Reload failed for HAProxy Load Balancer.

/var/log/audit/audit.log in the amp:

type=SERVICE_START msg=audit(1649425399.455:193): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=haproxy-482777a1-269c-4872-9a36-b883f08c1902 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'UID="root" AUID="unset"
type=AVC msg=audit(1649425422.841:194): avc:  denied  { entrypoint } for  pid=5633 comm="(sh)" path="/usr/bin/bash" dev="vda1" ino=4215617 scontext=system_u:system_r:haproxy_t:s0 tcontext=system_u:object_r:shell_exec_t:s0 tclass=file permissive=0


Version-Release number of selected component (if applicable):
17.0

How reproducible:
100%

Steps to Reproduce:
1. Create a LB, a listener, then create a pool, the amphora returns an error

Comment 3 Waldemar Znoinski 2022-04-11 12:57:06 UTC
hi Gregory

can this problem be seen/retested when running tests from one of the tempest/rally/tobiko frameworks do you know?

Comment 4 Gregory Thiemonge 2022-04-11 13:13:48 UTC
(In reply to Waldemar Znoinski from comment #3)
> hi Gregory
> 
> can this problem be seen/retested when running tests from one of the
> tempest/rally/tobiko frameworks do you know?

Yes, we can use the octavia-tempest-plugin to test it.
Sadly it is complicated to detect the issues because the selinux problems occur in a service VM created by Octavia and we don't have direct access or export to audit.log

Comment 8 Cédric Jeanneret 2022-04-13 13:57:39 UTC
Actual fix: https://review.opendev.org/c/openstack/octavia/+/837721

Comment 9 Gregory Thiemonge 2022-04-20 16:41:03 UTC
Backport proposed on stable/wallaby

Comment 11 Omer Schwartz 2022-05-18 13:02:15 UTC
The Octavia OSP17 jobs run on RHEL9, so as the following build

https://rhos-ci-jenkins.lab.eng.tlv2.redhat.com/view/DFG/view/network/view/octavia/job/DFG-network-octavia-17.0_director-rhel-virthost-3cont_3comp-ipv4-geneve-actstby/36/testReport/

which was run with the RHOS-17.0-RHEL-9-20220511.n.1 puddle, contains tests which show that the fix works, I am moving this BZ to VERIFIED.

Comment 16 errata-xmlrpc 2022-09-21 12:20:36 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Release of components for Red Hat OpenStack Platform 17.0 (Wallaby)), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2022:6543