Load balancer goes into ERROR on listener create. The error message in the Worker service hints that something went wrong in the amphora. Looking to the journal of the amphora instance, one can see warnings of NetworkManager not being able to open file /etc/sysconfig/network-scripts/ifcfg-eth1 and mount permission denied. These are due to SELinux AVCs.


Version-Release number of selected component (if applicable):
RHOS_TRUNK-16.0-RHEL-8-20191007.n.0

How reproducible:
100%

Steps to Reproduce:
1. Create load balancer and wait for ACTIVE operational status
2. Create listener (TCP:80 is good enough)
3. Observe load balancer and listener go into ERROR

Actual results:

Oct 26 11:06:20 amphora-cb8ebf5c-0927-4675-8048-5cea193a0fcc awk[1640]: WARN      : [ifup] You are using 'ifup' script provided by 'network-scripts', which are now deprecated.
Oct 26 11:06:20 amphora-cb8ebf5c-0927-4675-8048-5cea193a0fcc ifup[1649]: You are using 'ifup' script provided by 'network-scripts', which are now deprecated.
Oct 26 11:06:20 amphora-cb8ebf5c-0927-4675-8048-5cea193a0fcc awk[1640]: WARN      : [ifup] 'network-scripts' will be removed in one of the next major releases of RHEL.
Oct 26 11:06:20 amphora-cb8ebf5c-0927-4675-8048-5cea193a0fcc ifup[1650]: 'network-scripts' will be removed in one of the next major releases of RHEL.
Oct 26 11:06:20 amphora-cb8ebf5c-0927-4675-8048-5cea193a0fcc awk[1640]: WARN      : [ifup] It is advised to switch to 'NetworkManager' instead - it provides 'ifup/ifdown' scripts as well.
Oct 26 11:06:20 amphora-cb8ebf5c-0927-4675-8048-5cea193a0fcc ifup[1651]: It is advised to switch to 'NetworkManager' instead - it provides 'ifup/ifdown' scripts as well.
Oct 26 11:06:20 amphora-cb8ebf5c-0927-4675-8048-5cea193a0fcc NetworkManager[780]: <warn>  [1572102380.2425] ifcfg-rh: loading "/etc/sysconfig/network-scripts/ifcfg-eth1" fails: Could not read file '/etc/sysconfig/network-scripts/ifcfg-eth1': No such file or directory
Oct 26 11:06:20 amphora-cb8ebf5c-0927-4675-8048-5cea193a0fcc NetworkManager[780]: <warn>  [1572102380.3765] ifcfg-rh: loading "/etc/sysconfig/network-scripts/ifcfg-eth1" fails: Could not read file '/etc/sysconfig/network-scripts/ifcfg-eth1': No such file or directory
Oct 26 11:06:20 amphora-cb8ebf5c-0927-4675-8048-5cea193a0fcc NetworkManager[780]: <warn>  [1572102380.6253] ifcfg-rh: loading "/etc/sysconfig/network-scripts/ifcfg-eth1" fails: Could not read file '/etc/sysconfig/network-scripts/ifcfg-eth1': No such file or directory
Oct 26 11:06:20 amphora-cb8ebf5c-0927-4675-8048-5cea193a0fcc awk[1640]: ERROR     : [/etc/sysconfig/network-scripts/ifup-ipv6] Global IPv6 forwarding is disabled in configuration, but not currently disabled in kernel
Oct 26 11:06:20 amphora-cb8ebf5c-0927-4675-8048-5cea193a0fcc /etc/sysconfig/network-scripts/ifup-ipv6[1697]: Global IPv6 forwarding is disabled in configuration, but not currently disabled in kernel
Oct 26 11:06:20 amphora-cb8ebf5c-0927-4675-8048-5cea193a0fcc awk[1640]: ERROR     : [/etc/sysconfig/network-scripts/ifup-ipv6] Please restart network with '/sbin/service network restart'
Oct 26 11:06:20 amphora-cb8ebf5c-0927-4675-8048-5cea193a0fcc /etc/sysconfig/network-scripts/ifup-ipv6[1698]: Please restart network with '/sbin/service network restart'
Oct 26 11:06:20 amphora-cb8ebf5c-0927-4675-8048-5cea193a0fcc NetworkManager[780]: <warn>  [1572102380.9332] ifcfg-rh: loading "/etc/sysconfig/network-scripts/ifcfg-eth1" fails: Could not read file '/etc/sysconfig/network-scripts/ifcfg-eth1': No such file or directory
Oct 26 11:06:21 amphora-cb8ebf5c-0927-4675-8048-5cea193a0fcc systemd[1]: Started Configure amphora-haproxy network namespace.
Oct 26 11:06:21 amphora-cb8ebf5c-0927-4675-8048-5cea193a0fcc systemd[1]: Starting HAProxy Load Balancer...
Oct 26 11:06:21 amphora-cb8ebf5c-0927-4675-8048-5cea193a0fcc ip[1737]: mount of /sys failed: Permission denied
Oct 26 11:06:21 amphora-cb8ebf5c-0927-4675-8048-5cea193a0fcc systemd[1]: haproxy-1c37e27e-914a-4e44-a30e-8814e03f04f2.service: Main process exited, code=exited, status=1/FAILURE
Oct 26 11:06:21 amphora-cb8ebf5c-0927-4675-8048-5cea193a0fcc systemd[1]: haproxy-1c37e27e-914a-4e44-a30e-8814e03f04f2.service: Failed with result 'exit-code'.
Oct 26 11:06:21 amphora-cb8ebf5c-0927-4675-8048-5cea193a0fcc systemd[1]: Failed to start HAProxy Load Balancer.
Oct 26 11:06:21 amphora-cb8ebf5c-0927-4675-8048-5cea193a0fcc amphora-agent[1079]: 2019-10-26 11:06:21.408 1079 DEBUG octavia.amphorae.backends.agent.api_server.loadbalancer [-] Failed to start haproxy-1c37e27e-914a-4e44-a30e-8814e03f04f2 
service: Command '['/usr/sbin/service', 'haproxy-1c37e27e-914a-4e44-a30e-8814e03f04f2', 'start']' returned non-zero exit status 1. b'Redirecting to /bin/systemctl start haproxy-1c37e27e-914a-4e44-a30e-8814e03f04f2.service\nJob for haproxy
-1c37e27e-914a-4e44-a30e-8814e03f04f2.service failed because the control process exited with error code.\nSee "systemctl status haproxy-1c37e27e-914a-4e44-a30e-8814e03f04f2.service" and "journalctl -xe" for details.\n' start_stop_lb /usr/
lib/python3.6/site-packages/octavia/amphorae/backends/agent/api_server/loadbalancer.py:261
Oct 26 11:06:21 amphora-cb8ebf5c-0927-4675-8048-5cea193a0fcc amphora-agent[990]: 2019-10-26 11:06:21.408 1079 DEBUG octavia.amphorae.backends.agent.api_server.loadbalancer [-] Failed to start haproxy-1c37e27e-914a-4e44-a30e-8814e03f04f2 s
ervice: Command '['/usr/sbin/service', 'haproxy-1c37e27e-914a-4e44-a30e-8814e03f04f2', 'start']' returned non-zero exit status 1. b'Redirecting to /bin/systemctl start haproxy-1c37e27e-914a-4e44-a30e-8814e03f04f2.service\nJob for haproxy-
1c37e27e-914a-4e44-a30e-8814e03f04f2.service failed because the control process exited with error code.\nSee "systemctl status haproxy-1c37e27e-914a-4e44-a30e-8814e03f04f2.service" and "journalctl -xe" for details.\n' start_stop_lb /usr/l
ib/python3.6/site-packages/octavia/amphorae/backends/agent/api_server/loadbalancer.py:261
Oct 26 11:06:21 amphora-cb8ebf5c-0927-4675-8048-5cea193a0fcc systemd[1]: haproxy-1c37e27e-914a-4e44-a30e-8814e03f04f2.service: Service RestartSec=100ms expired, scheduling restart.
Oct 26 11:06:21 amphora-cb8ebf5c-0927-4675-8048-5cea193a0fcc systemd[1]: haproxy-1c37e27e-914a-4e44-a30e-8814e03f04f2.service: Scheduled restart job, restart counter is at 1.
Oct 26 11:06:21 amphora-cb8ebf5c-0927-4675-8048-5cea193a0fcc systemd[1]: Stopped HAProxy Load Balancer.
Oct 26 11:06:21 amphora-cb8ebf5c-0927-4675-8048-5cea193a0fcc systemd[1]: Starting HAProxy Load Balancer...
Oct 26 11:06:21 amphora-cb8ebf5c-0927-4675-8048-5cea193a0fcc ip[1742]: mount of /sys failed: Permission dThe error is caused by an eenied
Oct 26 11:06:21 amphora-cb8ebf5c-0927-4675-8048-5cea193a0fcc systemd[1]: haproxy-1c37e27e-914a-4e44-a30e-8814e03f04f2.service: Main process exited, code=exited, status=1/FAILURE
Oct 26 11:06:21 amphora-cb8ebf5c-0927-4675-8048-5cea193a0fcc systemd[1]: haproxy-1c37e27e-914a-4e44-a30e-8814e03f04f2.service: Failed with result 'exit-code'.
Oct 26 11:06:21 amphora-cb8ebf5c-0927-4675-8048-5cea193a0fcc systemd[1]: Failed to start HAProxy Load Balancer.
Oct 26 11:06:22 amphora-cb8ebf5c-0927-4675-8048-5cea193a0fcc systemd[1]: haproxy-1c37e27e-914a-4e44-a30e-8814e03f04f2.service: Service RestartSec=100ms expired, scheduling restart.
Oct 26 11:06:22 amphora-cb8ebf5c-0927-4675-8048-5cea193a0fcc systemd[1]: haproxy-1c37e27e-914a-4e44-a30e-8814e03f04f2.service: Scheduled restart job, restart counter is at 2.
Oct 26 11:06:22 amphora-cb8ebf5c-0927-4675-8048-5cea193a0fcc systemd[1]: Stopped HAProxy Load Balancer.
Oct 26 11:06:22 amphora-cb8ebf5c-0927-4675-8048-5cea193a0fcc systemd[1]: Starting HAProxy Load Balancer...
Oct 26 11:06:22 amphora-cb8ebf5c-0927-4675-8048-5cea193a0fcc ip[1745]: mount of /sys failed: Permission denied
Oct 26 11:06:22 amphora-cb8ebf5c-0927-4675-8048-5cea193a0fcc systemd[1]: haproxy-1c37e27e-914a-4e44-a30e-8814e03f04f2.service: Main process exited, code=exited, status=1/FAILURE
Oct 26 11:06:22 amphora-cb8ebf5c-0927-4675-8048-5cea193a0fcc systemd[1]: haproxy-1c37e27e-914a-4e44-a30e-8814e03f04f2.service: Failed with result 'exit-code'.
Oct 26 11:06:22 amphora-cb8ebf5c-0927-4675-8048-5cea193a0fcc systemd[1]: Failed to start HAProxy Load Balancer.
Oct 26 11:06:22 amphora-cb8ebf5c-0927-4675-8048-5cea193a0fcc systemd[1]: haproxy-1c37e27e-914a-4e44-a30e-8814e03f04f2.service: Service RestartSec=100ms expired, scheduling restart.
Oct 26 11:06:22 amphora-cb8ebf5c-0927-4675-8048-5cea193a0fcc systemd[1]: haproxy-1c37e27e-914a-4e44-a30e-8814e03f04f2.service: Scheduled restart job, restart counter is at 3.
Oct 26 11:06:22 amphora-cb8ebf5c-0927-4675-8048-5cea193a0fcc systemd[1]: Stopped HAProxy Load Balancer.
Oct 26 11:06:22 amphora-cb8ebf5c-0927-4675-8048-5cea193a0fcc systemd[1]: Starting HAProxy Load Balancer...
Oct 26 11:06:22 amphora-cb8ebf5c-0927-4675-8048-5cea193a0fcc ip[1752]: mount of /sys failed: Permission denied
Oct 26 11:06:22 amphora-cb8ebf5c-0927-4675-8048-5cea193a0fcc systemd[1]: haproxy-1c37e27e-914a-4e44-a30e-8814e03f04f2.service: Main process exited, code=exited, status=1/FAILURE
Oct 26 11:06:22 amphora-cb8ebf5c-0927-4675-8048-5cea193a0fcc systemd[1]: haproxy-1c37e27e-914a-4e44-a30e-8814e03f04f2.service: Failed with result 'exit-code'.
Oct 26 11:06:22 amphora-cb8ebf5c-0927-4675-8048-5cea193a0fcc systemd[1]: Failed to start HAProxy Load Balancer.
Oct 26 11:06:23 amphora-cb8ebf5c-0927-4675-8048-5cea193a0fcc systemd[1]: haproxy-1c37e27e-914a-4e44-a30e-8814e03f04f2.service: Service RestartSec=100ms expired, scheduling restart.
Oct 26 11:06:23 amphora-cb8ebf5c-0927-4675-8048-5cea193a0fcc systemd[1]: haproxy-1c37e27e-914a-4e44-a30e-8814e03f04f2.service: Scheduled restart job, restart counter is at 4.
Oct 26 11:06:23 amphora-cb8ebf5c-0927-4675-8048-5cea193a0fcc systemd[1]: Stopped HAProxy Load Balancer.
Oct 26 11:06:23 amphora-cb8ebf5c-0927-4675-8048-5cea193a0fcc systemd[1]: Starting HAProxy Load Balancer...
Oct 26 11:06:23 amphora-cb8ebf5c-0927-4675-8048-5cea193a0fcc ip[1755]: mount of /sys failed: Permission denied
Oct 26 11:06:23 amphora-cb8ebf5c-0927-4675-8048-5cea193a0fcc systemd[1]: haproxy-1c37e27e-914a-4e44-a30e-8814e03f04f2.service: Main process exited, code=exited, status=1/FAILURE
Oct 26 11:06:23 amphora-cb8ebf5c-0927-4675-8048-5cea193a0fcc systemd[1]: haproxy-1c37e27e-914a-4e44-a30e-8814e03f04f2.service: Failed with result 'exit-code'.
Oct 26 11:06:23 amphora-cb8ebf5c-0927-4675-8048-5cea193a0fcc systemd[1]: Failed to start HAProxy Load Balancer.
Oct 26 11:06:23 amphora-cb8ebf5c-0927-4675-8048-5cea193a0fcc systemd[1]: haproxy-1c37e27e-914a-4e44-a30e-8814e03f04f2.service: Service RestartSec=100ms expired, scheduling restart.
Oct 26 11:06:23 amphora-cb8ebf5c-0927-4675-8048-5cea193a0fcc systemd[1]: haproxy-1c37e27e-914a-4e44-a30e-8814e03f04f2.service: Scheduled restart job, restart counter is at 5.
Oct 26 11:06:23 amphora-cb8ebf5c-0927-4675-8048-5cea193a0fcc systemd[1]: Stopped HAProxy Load Balancer.
Oct 26 11:06:23 amphora-cb8ebf5c-0927-4675-8048-5cea193a0fcc systemd[1]: haproxy-1c37e27e-914a-4e44-a30e-8814e03f04f2.service: Start request repeated too quickly.
Oct 26 11:06:23 amphora-cb8ebf5c-0927-4675-8048-5cea193a0fcc systemd[1]: haproxy-1c37e27e-914a-4e44-a30e-8814e03f04f2.service: Failed with result 'exit-code'.
Oct 26 11:06:23 amphora-cb8ebf5c-0927-4675-8048-5cea193a0fcc systemd[1]: Failed to start HAProxy Load Balancer.
Oct 26 11:06:23 amphora-cb8ebf5c-0927-4675-8048-5cea193a0fcc systemd[1]: amphora-netns.service: Unit not needed anymore. Stopping.
Oct 26 11:06:23 amphora-cb8ebf5c-0927-4675-8048-5cea193a0fcc systemd[1]: Stopped Configure amphora-haproxy network namespace.


File /var/log/audit/audit.log:

type=AVC msg=audit(1572103537.454:113): avc:  denied  { read } for  pid=1752 comm="ip" dev="nsfs" ino=4026531992 scontext=system_u:system_r:haproxy_t:s0 tcontext=system_u:system_r:NetworkManager_t:s0 tclass=file permissive=1
type=AVC msg=audit(1572103537.454:113): avc:  denied  { open } for  pid=1752 comm="ip" path="net:[4026531992]" dev="nsfs" ino=4026531992 scontext=system_u:system_r:haproxy_t:s0 tcontext=system_u:system_r:NetworkManager_t:s0 tclass=file permissive=1