Bug 1872128

Summary: Can't run container with hostPort on ipv6 cluster
Product: OpenShift Container Platform Reporter: Nir <nyehia>
Component: NodeAssignee: Antonio Ojea <aojeagar>
Node sub component: CRI-O QA Contact: Weinan Liu <weinliu>
Status: CLOSED ERRATA Docs Contact:
Severity: medium    
Priority: medium CC: aojeagar, aos-bugs, dcbw, dwalsh, ealcaniz, jokerman, schoudha, tsweeney, weinliu, william.caban
Version: 4.5   
Target Milestone: ---   
Target Release: 4.7.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Cause: CRI-O only uses IPv4 iptables to implement containers Host Ports Consequence: HostPort doesn´t work for IPv6 Fix: Implement HostPort IPv6 support in CRI-O Result: CRI-O support HostPort for IPv6 and IPv4
 Story Points: ---
Clone Of:
: 1900658 (view as bug list) Environment:
Last Closed: 2021-02-24 15:16:08 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1897336, 1900658    

Description Nir 2020-08-25 07:00:20 UTC 
Description of problem:

While trying to deploy daemonset which is using hostPort I get the following when describing one of the pods:
  Normal   AddedInterface          2m8s                   multus                                       Add eth0 [fd01::6:e41e:24ff:fe00:a8/64]
  Warning  FailedCreatePodSandBox  2m7s (x5304 over 22h)  kubelet, worker-2.ostest.test.metalkube.org  (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to add hostport mapping for sandbox k8s_poison-pill-ds-nls79_default_6933bfce-b094-45d4-926d-9134a3d1cc82_0(11e205eec09420a90f965a5a43c616dfb37c236aae152d3e99798b69e5fc8eed): HostPortManager IP family mismatch: fd01::6:e41e:24ff:fe00:a8, isIPv6 - true
  Normal   AddedInterface          115s                   multus                                       Add eth0 [fd01::6:e41e:24ff:fe00:a8/64]
(many of these "Add eth0 .... " )


Version-Release number of selected component (if applicable):
$ oc version
Client Version: 4.5.0-0.ci-2020-08-12-000552
Server Version: 4.5.0-0.ci-2020-08-12-000552
Kubernetes Version: v1.18.3


How reproducible:
always

Steps to Reproduce:
1.deploy container with hostPort on ipv6 cluster

Actual results:
container stuck on `ContainerCreating` state, decribe pod shows the error mentioned above

Expected results:
Container will be in running state, and listening on the specified hostPort
Comment 1 Antonio Ojea 2020-08-25 08:03:00 UTC 
It seems that CRI-O is always using iptables IPv4 for managing the host port mapping

https://github.com/cri-o/cri-o/blob/81a23e1553271e747d177b863235a24337f41181/server/server.go#L338-L342

> 	iptInterface := utiliptables.New(utilexec.New(), utiliptables.ProtocolIPv4)

One solution to fix this bug would be to parse the network configuration and initializing the hostportManager with the proper ipFamily. However, this would not solve the dual stack problem, where we should run two hostPort managers in parallel, one for each ipFamily.

We solved this in kubernetes upstream https://github.com/kubernetes/kubernetes/pull/80854 using one portManager per IP family and leveraging the sandbox IPs to choose the right one.

I submitted a PR with the later approach https://github.com/cri-o/cri-o/pull/4116
Comment 2 Seth Jennings 2020-08-28 15:23:20 UTC 
cri-o PR merged
Comment 16 Weinan Liu 2020-12-09 15:31:02 UTC 
Blocked by [BM][IPI] Master deployment failed: No valid host was found. Reason: No conductor service registered which supports driver redfish for conductor group https://bugzilla.redhat.com/show_bug.cgi?id=1902653, we can not get an ipv6 cluster built
Comment 19 Antonio Ojea 2020-12-11 11:52:07 UTC 
Have you curled/pinged/polled the exposed hostPort to verify that it acually forwards the traffic to the container?
Comment 20 Weinan Liu 2020-12-18 07:51:49 UTC 
We only managed to get clusters of ipv6 & disconnected, no mirrored images available.
I checked pod creating with a Completed status.
Comment 23 errata-xmlrpc 2021-02-24 15:16:08 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:5633