Bug 1857672 - Failed to ensure HAProxy PREROUTING rule to direct traffic to the LB because of the incorrect iptables binaries path
Summary: Failed to ensure HAProxy PREROUTING rule to direct traffic to the LB because ...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Installer
Version: 4.6
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.6.0
Assignee: Beth White
QA Contact: Aleksandra Malykhin
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-07-16 10:27 UTC by Aleksandra Malykhin
Modified: 2020-10-27 16:15 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: In recent images there are iptables binaries in /usr/sbin/ in addition to/in place of /usr/bin. This is causing problems because /usr/sbin is in the PATH before /usr/bin. Consequence: Since the code creating the IPtables rules which redirect API traffic to HAProxy load balancer don't apply correctly API calls don't distribute between all master nodes. Fix: Copy the correct iptables command also to /usr/sbin/ Result: The API calls are distributed to all master nodes by the load balancer
Clone Of:
Environment:
Last Closed: 2020-10-27 16:15:06 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift baremetal-runtimecfg pull 72 0 None closed Bug 1857672: Also copy iptables scripts to /usr/sbin 2020-10-07 10:29:53 UTC
Red Hat Product Errata RHBA-2020:4196 0 None None None 2020-10-27 16:15:43 UTC

Description Aleksandra Malykhin 2020-07-16 10:27:52 UTC
Description of problem:

In recent images there are iptables binaries in /usr/sbin/ in
addition to/in place of /usr/bin. This is causing problems because
/usr/sbin is in the PATH before /usr/bin so our code is calling the
wrong iptables. 

How reproducible:

Steps to Reproduce:
1. Deploy the 4.6 OCP cluster
2. Open the haproxy-monitor logs

Actual results:
2020-07-15T13:14:18.004082109+00:00 stderr F time="2020-07-15T13:14:18Z" level=info msg="Inserting nat PREROUTING rule" spec="--dst fd2e:6f44:5dd8::5 -p tcp --dport 6443 -j REDIRECT --to-ports 9445 -m comment --comment OCP_API_LB_REDIRECT"
2020-07-15T13:14:18.007182811+00:00 stderr F time="2020-07-15T13:14:18Z" level=error msg="Failed to ensure HAProxy PREROUTING rule to direct traffic to the LB" err="running [/usr/sbin/ip6tables -t nat -I PREROUTING 1 --dst fd2e:6f44:5dd8::5 -p tcp --dport 6443 -j REDIRECT --to-ports 9445 -m comment --comment OCP_API_LB_REDIRECT --wait]: exit status 3: ip6tables v1.4.21: can't initialize ip6tables table `nat': Table does not exist (do you need to insmod?)\nPerhaps ip6tables or your kernel needs to be upgraded.\n"

Expected results:
No errors in the haproxy-monitor logs

Additional info:
Please attach logs from ansible-playbook with the -vvv flag

Comment 3 Aleksandra Malykhin 2020-09-03 06:17:03 UTC
Verified on the version ocp-release:4.6.0-fc.3-x86_64

1. Connect to the API VIP node $ ssh core@<API_VIP node>
2. Open haproxy-monitor log $ sudo cat /var/log/pods/openshift-kni-infra_haproxy-master-0-1_633505fb84bb6e6d52d64dd2f6aa893b/haproxy-monitor/0.log

No incorrect iptables binaries path error found in the haproxy-monitor logs
Ip6tables are located under "/usr/sbin" path

Comment 5 errata-xmlrpc 2020-10-27 16:15:06 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.6 GA Images), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4196


Note You need to log in before you can comment on or make changes to this bug.