Bug 2213742

Summary: TCP/HTTP health-monitors in UDP pools work only on specific ports
Product: Red Hat OpenStack Reporter: Gregory Thiemonge <gthiemon>
Component: openstack-octaviaAssignee: Gregory Thiemonge <gthiemon>
Status: MODIFIED --- QA Contact: Bruna Bonguardo <bbonguar>
Severity: medium Docs Contact: Greg Rakauskas <gregraka>
Priority: medium    
Version: 17.1 (Wallaby)CC: enothen, gthiemon, tweining
Target Milestone: z2Keywords: Triaged
Target Release: 17.1Flags: ifrangs: needinfo? (gthiemon)
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-octavia-12.0.1-18.0.20230713233746.3ac52a0.el9ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Gregory Thiemonge 2023-06-09 06:08:24 UTC
Description of problem:
With OSP17, users can add TCP/HTTP health-monitors to UDP pools, but some ports are denied by SELinux

Version-Release number of selected component (if applicable):
17.1

How reproducible:
100%

Steps to Reproduce:
1. Deploy OSP+Octavia
2. Create a LB+UDP Listener+UDP Pool+1 member (port 1234 or monitor_port 1234)+TCP or HTTP HM
3. 

Actual results:
No TCP/HTTP packets are sent by keepalived (and there are SELinux denials in the logs)
HM doesn't work

Expected results:
HM packets should not be blocked

Additional info:

Reproducer:

openstack loadbalancer create --vip-subnet external_subnet --name lb1
openstack loadbalancer listener create --protocol UDP --protocol-port 1234 --name listener1 lb1
openstack loadbalancer pool create --protocol UDP --listener listener1 --lb-algorithm ROUND_ROBIN --name pool1
openstack loadbalancer member create --address 10.0.0.72 --protocol-port 1234 --name member1 pool1
openstack loadbalancer healthmonitor create --name hm1 --type HTTP --delay 5 --timeout 4 --max-retries 3 pool1


Log into the amphora, check the audit log:

# tail -f /var/log/audit/audit.log
type=AVC msg=audit(1686240221.266:838): avc:  denied  { name_connect } for  pid=15945 comm="keepalived" dest=1234 scontext=system_u:system_r:keepalived_t:s0 tcontext=system_u:object_r:monopd_port_t:s0 tclass=tcp_socket permissive=0
type=SYSCALL msg=audit(1686240221.266:838): arch=c000003e syscall=42 success=no exit=-13 a0=8 a1=555c80b97600 a2=80 a3=7ffcdc746b50 items=0 ppid=15944 pid=15945 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="keepalived" exe="/usr/sbin/keepalived" subj=system_u:system_r:keepalived_t:s0 key=(null)ARCH=x86_64 SYSCALL=connect AUID="unset" UID="root" GID="root" EUID="root" SUID="root" FSUID="root" EGID="root" SGID="root" FSGID="root"

Check tcpdump output:

# ip netns exec amphora-haproxy tcpdump -nn -i eth1
[..] no TCP packets


Try another port number:

openstack loadbalancer member set --monitor-port 1235 pool1 member1

Check the logs and tcpdump output:

# tail -f /var/log/audit/audit.log
[..] no denied for name_connect

# ip netns exec amphora-haproxy tcpdump -nn -i eth1
[..]
12:06:22.175945 IP 10.0.0.233.18583 > 10.0.0.72.1235: Flags [S], seq 1796162379, win 29200, options [mss 1460,sackOK,TS val 3659440612 ecr 0,nop,wscale 11], length 0
[..]


It seems that keepalived allows only TCP connections to a specific list of port numbers:

There's a related BZ for RHEL7 https://bugzilla.redhat.com/show_bug.cgi?id=1443473
Related KCS https://access.redhat.com/solutions/2119681


It can be mitigated with a SELinux boolean:

# semanage boolean -l | grep keepal
keepalived_connect_any         (off  ,  off)  Allow keepalived to connect any

From the keepalived_selinux manpage:

| If you want to determine whether keepalived can connect to all TCP ports, you must turn on the keepalived_connect_any boolean. Disabled by default.
| setsebool -P keepalived_connect_any 1

Comment 6 Gregory Thiemonge 2023-06-14 08:42:49 UTC
Fix proposed to octavia master

Comment 8 Gregory Thiemonge 2023-07-03 08:28:51 UTC
Backport proposed to stable/wallaby