Bug 1857672

Summary: Failed to ensure HAProxy PREROUTING rule to direct traffic to the LB because of the incorrect iptables binaries path
Product: OpenShift Container Platform Reporter: Aleksandra Malykhin <amalykhi>
Component: InstallerAssignee: Beth White <beth.white>
Installer sub component: OpenShift on Bare Metal IPI QA Contact: Aleksandra Malykhin <amalykhi>
Status: CLOSED ERRATA Docs Contact:
Severity: high    
Priority: high CC: bnemec, bperkins, yboaron
Version: 4.6Keywords: Triaged
Target Milestone: ---   
Target Release: 4.6.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Cause: In recent images there are iptables binaries in /usr/sbin/ in addition to/in place of /usr/bin. This is causing problems because /usr/sbin is in the PATH before /usr/bin. Consequence: Since the code creating the IPtables rules which redirect API traffic to HAProxy load balancer don't apply correctly API calls don't distribute between all master nodes. Fix: Copy the correct iptables command also to /usr/sbin/ Result: The API calls are distributed to all master nodes by the load balancer
 Story Points: ---
Clone Of: Environment:
Last Closed: 2020-10-27 16:15:06 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Aleksandra Malykhin 2020-07-16 10:27:52 UTC 
Description of problem:

In recent images there are iptables binaries in /usr/sbin/ in
addition to/in place of /usr/bin. This is causing problems because
/usr/sbin is in the PATH before /usr/bin so our code is calling the
wrong iptables. 

How reproducible:

Steps to Reproduce:
1. Deploy the 4.6 OCP cluster
2. Open the haproxy-monitor logs

Actual results:
2020-07-15T13:14:18.004082109+00:00 stderr F time="2020-07-15T13:14:18Z" level=info msg="Inserting nat PREROUTING rule" spec="--dst fd2e:6f44:5dd8::5 -p tcp --dport 6443 -j REDIRECT --to-ports 9445 -m comment --comment OCP_API_LB_REDIRECT"
2020-07-15T13:14:18.007182811+00:00 stderr F time="2020-07-15T13:14:18Z" level=error msg="Failed to ensure HAProxy PREROUTING rule to direct traffic to the LB" err="running [/usr/sbin/ip6tables -t nat -I PREROUTING 1 --dst fd2e:6f44:5dd8::5 -p tcp --dport 6443 -j REDIRECT --to-ports 9445 -m comment --comment OCP_API_LB_REDIRECT --wait]: exit status 3: ip6tables v1.4.21: can't initialize ip6tables table `nat': Table does not exist (do you need to insmod?)\nPerhaps ip6tables or your kernel needs to be upgraded.\n"

Expected results:
No errors in the haproxy-monitor logs

Additional info:
Please attach logs from ansible-playbook with the -vvv flag
Comment 3 Aleksandra Malykhin 2020-09-03 06:17:03 UTC 
Verified on the version ocp-release:4.6.0-fc.3-x86_64

1. Connect to the API VIP node $ ssh core@<API_VIP node>
2. Open haproxy-monitor log $ sudo cat /var/log/pods/openshift-kni-infra_haproxy-master-0-1_633505fb84bb6e6d52d64dd2f6aa893b/haproxy-monitor/0.log

No incorrect iptables binaries path error found in the haproxy-monitor logs
Ip6tables are located under "/usr/sbin" path
Comment 5 errata-xmlrpc 2020-10-27 16:15:06 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.6 GA Images), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4196