Bug 1915027 - fix MCS blocking iptables rules
Summary: fix MCS blocking iptables rules
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Networking
Version: 4.7
Hardware: Unspecified
OS: Unspecified
urgent
high
Target Milestone: ---
: 4.7.0
Assignee: Dan Winship
QA Contact: Ying Wang
URL:
Whiteboard:
Depends On:
Blocks: 1937118 1921274 1998453
TreeView+ depends on / blocked
 
Reported: 2021-01-11 19:11 UTC by Dan Winship
Modified: 2021-09-23 18:25 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: IPTables rules designed to block pods from connecting to a cluster's machine-config-server were being too aggressive and accidentally blocked any TCP connection involving port 22623 or 22624 on either side. (This affected both openshift-sdn and ovn-kubernetes.) Consequence: On rare occasions, an inbound or outbound TCP connection would fail for no obvious reason, because the client side had randomly picked either 22623 or 22624 as the source port. Fix: The iptables rules were fixed to only block new outbound connections to ports 22623 or 22624, rather than all traffic to ports 22623 and 22624. Result: Random connections are not blocked.
Clone Of:
: 1998453 (view as bug list)
Environment:
Last Closed: 2021-02-24 15:51:51 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift ovn-kubernetes pull 400 0 None closed Bug 1915027: Fix MCS-blocking iptables rules 2021-02-08 12:03:13 UTC
Github openshift sdn pull 243 0 None closed Bug 1915027: Fix MCS-blocking iptables rules 2021-02-08 12:03:13 UTC
Red Hat Knowledge Base (Solution) 5709711 0 None None None 2021-01-18 23:54:36 UTC
Red Hat Product Errata RHSA-2020:5633 0 None None None 2021-02-24 15:52:13 UTC

Description Dan Winship 2021-01-11 19:11:47 UTC
The ovn-kubernetes and openshift-sdn rules to block access to the MCS were accidentally interfering with (replies to) connections that happened to randomly pick 22623 or 22624 as the source port.


To verify: try a command such as

  curl --local-port 22624 http://google.com/

from inside a pod. It should succeed (in this case, returning an HTTP 301 response) rather than hanging and eventually timing out.

(This bug is being used for both the ovn-kubernetes fix and the openshift-sdn fix and it should be tested on both.)

Comment 1 Dan Winship 2021-01-12 13:58:50 UTC
(In reply to Dan Winship from comment #0)
> To verify: try a command such as
> 
>   curl --local-port 22624 http://google.com/
> 
> from inside a pod. It should succeed (in this case, returning an HTTP 301
> response) rather than hanging and eventually timing out.

Hm... try

  curl --local-port 22624 https://172.30.0.1/

too. The "google.com" version apparently works with or without the fix, at least on openshift-sdn.

(Note that "curl --local-port 22624 https://172.30.0.1/" will return an error about the certificate being untrusted, which is fine; as long as it doesn't just hang and time out, it passes.)

Comment 7 Kedar Kulkarni 2021-01-27 20:31:13 UTC
Hi,
I was able to verify the BZ on OVN and SDN cluster using `curl` command to curl 172.30.0.1 as per the instructions. Here's the snippet of one of the curl calls:

===========================================================================================
oc run -it --rm curlpod --image=curlimages/curl -- /bin/sh
If you don't see a command prompt, try pressing enter.

/ $ curl --local-port 22624 https://172.30.0.1/
curl: (60) SSL certificate problem: self signed certificate in certificate chain
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
===========================================================================================

Nightly Build used: 4.7.0-0.nightly-2021-01-26-133503


Thanks,
KK.

Comment 10 errata-xmlrpc 2021-02-24 15:51:51 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:5633


Note You need to log in before you can comment on or make changes to this bug.