Description of problem: In OSP17 (RHEL8 and 9), Octavia fails to reload haproxy after each configuration update. The worker logs show: 2022-04-08 13:45:16.578 38 DEBUG octavia.controller.worker.v1.controller_worker [-] Task 'octavia.controller.worker.v1.tasks.amphora_driver_tasks.ListenersUpdate' (66ff05b8-0756-4e4d-85df-f09a71805b4b) transitioned into state 'RUNNING' from state 'PENDING' _task_receiver /usr/lib/python3.6/site-packages/taskflow/listeners/logging.py:192 2022-04-08 13:45:16.578 38 DEBUG octavia.amphorae.drivers.haproxy.rest_api_driver [req-627fe0bb-cbb1-491e-a682-ccf199cb577f - 2886febc2f0c44fea2250ec811834f37 - - -] request url / request /usr/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py:678 2022-04-08 13:45:16.579 38 DEBUG octavia.amphorae.drivers.haproxy.rest_api_driver [req-627fe0bb-cbb1-491e-a682-ccf199cb577f - 2886febc2f0c44fea2250ec811834f37 - - -] request url https://172.24.3.163:9443// request /usr/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py:681 2022-04-08 13:45:16.597 38 DEBUG octavia.amphorae.drivers.haproxy.rest_api_driver [req-627fe0bb-cbb1-491e-a682-ccf199cb577f - 2886febc2f0c44fea2250ec811834f37 - - -] Connected to amphora. Response: <Response [200]> request /usr/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py:702 2022-04-08 13:45:16.597 38 DEBUG octavia.amphorae.drivers.haproxy.rest_api_driver [req-627fe0bb-cbb1-491e-a682-ccf199cb577f - 2886febc2f0c44fea2250ec811834f37 - - -] Amphora 3d8868f5-088f-44e7-88b4-fe860f2f0972 has API version 1.0 _populate_amphora_api_version /usr/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py:112 2022-04-08 13:45:16.598 38 DEBUG octavia.amphorae.drivers.haproxy.rest_api_driver [req-627fe0bb-cbb1-491e-a682-ccf199cb577f - 2886febc2f0c44fea2250ec811834f37 - - -] request url info request /usr/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py:678 2022-04-08 13:45:16.598 38 DEBUG octavia.amphorae.drivers.haproxy.rest_api_driver [req-627fe0bb-cbb1-491e-a682-ccf199cb577f - 2886febc2f0c44fea2250ec811834f37 - - -] request url https://172.24.3.163:9443/1.0/info request /usr/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py:681 2022-04-08 13:45:16.663 38 DEBUG octavia.amphorae.drivers.haproxy.rest_api_driver [req-627fe0bb-cbb1-491e-a682-ccf199cb577f - 2886febc2f0c44fea2250ec811834f37 - - -] Connected to amphora. Response: <Response [200]> request /usr/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py:702 2022-04-08 13:45:16.663 38 DEBUG octavia.amphorae.drivers.haproxy.rest_api_driver [req-627fe0bb-cbb1-491e-a682-ccf199cb577f - 2886febc2f0c44fea2250ec811834f37 - - -] Amphora 3d8868f5-088f-44e7-88b4-fe860f2f0972 has API version 1.0 _populate_amphora_api_version /usr/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py:112 2022-04-08 13:45:16.664 38 DEBUG octavia.amphorae.drivers.haproxy.rest_api_driver [req-627fe0bb-cbb1-491e-a682-ccf199cb577f - 2886febc2f0c44fea2250ec811834f37 - - -] Amphora 3d8868f5-088f-44e7-88b4-fe860f2f0972 for loadbalancer 482777a1-269c-4872-9a36-b883f08c1902 is already in single process mode. update_amphora_listeners /usr/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py:150 2022-04-08 13:45:16.664 38 DEBUG octavia.amphorae.drivers.haproxy.rest_api_driver [req-627fe0bb-cbb1-491e-a682-ccf199cb577f - 2886febc2f0c44fea2250ec811834f37 - - -] HaproxyAmphoraLoadBalancerDriver updating listener 70ebb045-83de-47bc-ac39-46fd86c29f45 on amphora 3d8868f5-088f-44e7-88b4-fe860f2f0972 update_amphora_listeners /usr/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py:157 2022-04-08 13:45:16.665 38 DEBUG octavia.amphorae.drivers.haproxy.rest_api_driver [req-4ed5f5fa-ed1b-44b0-a48b-4868c2e7f52a - b770a0d5d13744fface4b7406fbc4805 - - -] request url loadbalancer/3d8868f5-088f-44e7-88b4-fe860f2f0972/482777a1-269c-4872-9a36-b883f08c1902/haproxy request /usr/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py:678 2022-04-08 13:45:16.665 38 DEBUG octavia.amphorae.drivers.haproxy.rest_api_driver [req-4ed5f5fa-ed1b-44b0-a48b-4868c2e7f52a - b770a0d5d13744fface4b7406fbc4805 - - -] request url https://172.24.3.163:9443/1.0/loadbalancer/3d8868f5-088f-44e7-88b4-fe860f2f0972/482777a1-269c-4872-9a36-b883f08c1902/haproxy request /usr/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py:681 2022-04-08 13:45:17.302 38 DEBUG octavia.amphorae.drivers.haproxy.rest_api_driver [req-4ed5f5fa-ed1b-44b0-a48b-4868c2e7f52a - b770a0d5d13744fface4b7406fbc4805 - - -] Connected to amphora. Response: <Response [202]> request /usr/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py:702 2022-04-08 13:45:17.303 38 DEBUG octavia.amphorae.drivers.haproxy.rest_api_driver [req-4ed5f5fa-ed1b-44b0-a48b-4868c2e7f52a - b770a0d5d13744fface4b7406fbc4805 - - -] request url loadbalancer/482777a1-269c-4872-9a36-b883f08c1902/reload request /usr/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py:678 2022-04-08 13:45:17.303 38 DEBUG octavia.amphorae.drivers.haproxy.rest_api_driver [req-4ed5f5fa-ed1b-44b0-a48b-4868c2e7f52a - b770a0d5d13744fface4b7406fbc4805 - - -] request url https://172.24.3.163:9443/1.0/loadbalancer/482777a1-269c-4872-9a36-b883f08c1902/reload request /usr/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py:681 2022-04-08 13:45:17.382 38 DEBUG octavia.amphorae.drivers.haproxy.rest_api_driver [req-4ed5f5fa-ed1b-44b0-a48b-4868c2e7f52a - b770a0d5d13744fface4b7406fbc4805 - - -] Connected to amphora. Response: <Response [500]> request /usr/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py:702 2022-04-08 13:45:17.382 38 ERROR octavia.amphorae.drivers.haproxy.exceptions [req-4ed5f5fa-ed1b-44b0-a48b-4868c2e7f52a - b770a0d5d13744fface4b7406fbc4805 - - -] Amphora agent returned unexpected result code 500 with response {'message': 'Error reloading haproxy', 'details': 'Redirecting to /bin/systemctl reload haproxy-482777a1-269c-4872-9a36-b883f08c1902.service\nJob for haproxy-482777a1-269c-4872-9a36-b883f08c1902.service failed.\nSee "systemctl status haproxy-482777a1-269c-4872-9a36-b883f08c1902.service" and "journalctl -xe" for details.\n'} 2022-04-08 13:45:17.385 38 WARNING octavia.controller.worker.v1.controller_worker [-] Task 'octavia.controller.worker.v1.tasks.amphora_driver_tasks.ListenersUpdate' (66ff05b8-0756-4e4d-85df-f09a71805b4b) transitioned into state 'FAILURE' from state 'RUNNING' In the amphora logs: Apr 08 09:45:17 amphora-3d8868f5-088f-44e7-88b4-fe860f2f0972 systemd[1]: Reloading HAProxy Load Balancer. Apr 08 09:45:17 amphora-3d8868f5-088f-44e7-88b4-fe860f2f0972 systemd[5808]: haproxy-482777a1-269c-4872-9a36-b883f08c1902.service: Failed to execute command: Permission denied Apr 08 09:45:17 amphora-3d8868f5-088f-44e7-88b4-fe860f2f0972 systemd[5808]: haproxy-482777a1-269c-4872-9a36-b883f08c1902.service: Failed at step EXEC spawning /bin/sh: Permission denied Apr 08 09:45:17 amphora-3d8868f5-088f-44e7-88b4-fe860f2f0972 systemd[1]: haproxy-482777a1-269c-4872-9a36-b883f08c1902.service: Control process exited, code=exited status=203 Apr 08 09:45:17 amphora-3d8868f5-088f-44e7-88b4-fe860f2f0972 amphora-agent[1425]: 2022-04-08 09:45:17.126 1425 DEBUG octavia.amphorae.backends.agent.api_server.loadbalancer [-] Failed to reload haproxy-482777a1-269c-4872-9a36-b883f08c1902 service: Command '['/usr/sbin/service', 'haproxy-482777a1-269c-4872-9a36-b883f08c1902', 'reload']' returned non-zero exit status 1. b'Redirecting to /bin/systemctl reload haproxy-482777a1-269c-4872-9a36-b883f08c1902.service\nJob for haproxy-482777a1-269c-4872-9a36-b883f08c1902.service failed.\nSee "systemctl status haproxy-482777a1-269c-4872-9a36-b883f08c1902.service" and "journalctl -xe" for details.\n' start_stop_lb /usr/lib/python3.6/site-packages/octavia/amphorae/backends/agent/api_server/loadbalancer.py:258 Apr 08 09:45:17 amphora-3d8868f5-088f-44e7-88b4-fe860f2f0972 amphora-agent[1066]: 2022-04-08 09:45:17.126 1425 DEBUG octavia.amphorae.backends.agent.api_server.loadbalancer [-] Failed to reload haproxy-482777a1-269c-4872-9a36-b883f08c1902 service: Command '['/usr/sbin/service', 'haproxy-482777a1-269c-4872-9a36-b883f08c1902', 'reload']' returned non-zero exit status 1. b'Redirecting to /bin/systemctl reload haproxy-482777a1-269c-4872-9a36-b883f08c1902.service\nJob for haproxy-482777a1-269c-4872-9a36-b883f08c1902.service failed.\nSee "systemctl status haproxy-482777a1-269c-4872-9a36-b883f08c1902.service" and "journalctl -xe" for details.\n' start_stop_lb /usr/lib/python3.6/site-packages/octavia/amphorae/backends/agent/api_server/loadbalancer.py:258 Apr 08 09:45:17 amphora-3d8868f5-088f-44e7-88b4-fe860f2f0972 systemd[1]: Reload failed for HAProxy Load Balancer. /var/log/audit/audit.log in the amp: type=SERVICE_START msg=audit(1649425399.455:193): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=haproxy-482777a1-269c-4872-9a36-b883f08c1902 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'UID="root" AUID="unset" type=AVC msg=audit(1649425422.841:194): avc: denied { entrypoint } for pid=5633 comm="(sh)" path="/usr/bin/bash" dev="vda1" ino=4215617 scontext=system_u:system_r:haproxy_t:s0 tcontext=system_u:object_r:shell_exec_t:s0 tclass=file permissive=0 Version-Release number of selected component (if applicable): 17.0 How reproducible: 100% Steps to Reproduce: 1. Create a LB, a listener, then create a pool, the amphora returns an error
hi Gregory can this problem be seen/retested when running tests from one of the tempest/rally/tobiko frameworks do you know?
(In reply to Waldemar Znoinski from comment #3) > hi Gregory > > can this problem be seen/retested when running tests from one of the > tempest/rally/tobiko frameworks do you know? Yes, we can use the octavia-tempest-plugin to test it. Sadly it is complicated to detect the issues because the selinux problems occur in a service VM created by Octavia and we don't have direct access or export to audit.log
Actual fix: https://review.opendev.org/c/openstack/octavia/+/837721
Backport proposed on stable/wallaby
The Octavia OSP17 jobs run on RHEL9, so as the following build https://rhos-ci-jenkins.lab.eng.tlv2.redhat.com/view/DFG/view/network/view/octavia/job/DFG-network-octavia-17.0_director-rhel-virthost-3cont_3comp-ipv4-geneve-actstby/36/testReport/ which was run with the RHOS-17.0-RHEL-9-20220511.n.1 puddle, contains tests which show that the fix works, I am moving this BZ to VERIFIED.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Release of components for Red Hat OpenStack Platform 17.0 (Wallaby)), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2022:6543