Bug 1729603 - port forwarding to containers run via podman does not appear to work
Summary: port forwarding to containers run via podman does not appear to work
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Containers
Version: 4.2.0
Hardware: Unspecified
OS: Unspecified
low
low
Target Milestone: ---
: 4.6.0
Assignee: Matthew Heon
QA Contact: weiwei jiang
URL:
Whiteboard:
: 1747429 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-07-12 19:44 UTC by Micah Abbott
Modified: 2020-05-27 17:28 UTC (History)
17 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1762970 (view as bug list)
Environment:
Last Closed: 2020-05-27 17:28:47 UTC
Target Upstream Version:


Attachments (Terms of Use)

Comment 22 Daniel Walsh 2019-08-14 12:20:34 UTC
https://github.com/containers/libpod/pull/3755 is working on this.

Comment 33 Daniel Walsh 2019-09-04 12:47:56 UTC
*** Bug 1747429 has been marked as a duplicate of this bug. ***

Comment 34 Christophe Fergeau 2019-09-23 08:13:34 UTC
Any reason why all the discussion here was private? We'd like to link to the discussion in that bug in upstream code, but this bug is useless to !rh employees. Can we make all these comments public?

Comment 35 Matthew Heon 2019-09-23 13:11:12 UTC
We may wish to scrub the debug logs, but I don't think anything else would be sensitive.

In a brief summary, the issue is cause by conflicts between the Podman and CRI-O default CNI networks - the CRI-O network cannot successfully configure Podman networks. Normally this is not a problem, as Podman explicitly requests the Podman network be used for its containers; however, Podman 1.4.x appears to have a bug where this requested default is ignored, and the first network located (the CRI-O network) is used. Podman 1.5.x appears to have fixed the issue, but Openshift 4.2 and RHEL 8.1 are shipping Podman 1.4.2.

As a workaround, removing the two CRI-O CNI networks in /etc/cni/net.d/ (100-crio-bridge.conf and 200-loopback.conf) should work fine for Openshift nodes - CRI-O networking is configured elsewhere, so the default networks are not used.

Comment 37 Praveen Kumar 2020-02-17 06:53:12 UTC
@Matthew, mcambria@redhat.com Is this bug is fixed with latest version of podman which now available in 4.3 release? @Micah Any testing done against 4.3 side, we are still using the workaround which @Matthew suggested https://github.com/code-ready/snc/blob/master/createdisk.sh#L341-L345 and now facing a different issue https://bugzilla.redhat.com/show_bug.cgi?id=1803635, I hope this workaround is not related to this issue.

Comment 38 Matthew Heon 2020-02-17 14:20:24 UTC
It may be. I'll tag in Peter Hunt from the CRI-O team, as I believe the resolution (removing the unused, conflicting config files) was in their packaging.

Comment 39 Micah Abbott 2020-02-17 18:48:47 UTC
@Praveen

Testing on RHCOS 4.4 with default settings shows that we are not encountering this problem any more.

```
$ rpm-ostree status
State: idle
AutomaticUpdates: disabled
Deployments:
* ostree://a6ebea6e29826f7f619b0ecde7dd5c4c3ec5dc23c1d73393cfcf3a7e958478bd
                   Version: 44.81.202002141514-0 (2020-02-14T15:20:58Z)

$ rpm -q podman cri-o
podman-1.6.4-2.module+el8.1.1+5363+bf8ff1af.x86_64
cri-o-1.17.0-4.dev.rhaos4.4.gitc3436cc.el8.x86_64

$ sudo podman run -d -p 8080:80 -v /var/tmp/index.html:/usr/share/nginx/html/index.html:z docker.io/library/nginx    
19c86517c3ff781d29ffcf9ed8fe8d1b51598557b80629a9fb087971e6468727
[core@coreos ~]$ curl --max-time 10 -4 -vvv http://localhost:8080
* Rebuilt URL to: http://localhost:8080/             
*   Trying 127.0.0.1...
* TCP_NODELAY set          
* Connected to localhost (127.0.0.1) port 8080 (#0)
> GET / HTTP/1.1
> Host: localhost:8080    
> User-Agent: curl/7.61.1
> Accept: */*                                                                                             
>                                                                                                         
< HTTP/1.1 200 OK
< Server: nginx/1.17.8
< Date: Mon, 17 Feb 2020 18:45:46 GMT
< Content-Type: text/html
< Content-Length: 6
< Last-Modified: Mon, 17 Feb 2020 18:44:00 GMT
< Connection: keep-alive
< ETag: "5e4adef0-6"
< Accept-Ranges: bytes
< 
hello
* Connection #0 to host localhost left intact
```

Additionally, RHCOS 4.3 with default settings is also successful:

```
$ rpm-ostree status
rState: idle
AutomaticUpdates: disabled
Deployments:
* ostree://08ee6458b08e6e0a5b773ecd189cf842b58d685ff8eda88d241a7ad54f22aad2
                   Version: 43.81.202002110953.0 (2020-02-11T09:59:02Z)

$ rpm -q cri-o podman
cri-o-1.16.3-19.dev.rhaos4.3.git6c1f4bd.el8.x86_64
podman-1.6.4-2.module+el8.1.1+5363+bf8ff1af.x86_64

$ sudo podman run -d -p 8080:80 -v /var/tmp/index.html:/usr/share/nginx/html/index.html:z docker.io/library/nginx  
Trying to pull docker.io/library/nginx...
Getting image source signatures
Copying blob bf317aa10aa5 done
Copying blob bc51dd8edc1b done
Copying blob 66ba67045f57 done
Copying config 2073e0bcb6 done
Writing manifest to image destination
Storing signatures
7a2d31d699b67d89223a48c81ec29ee54884d10a3e44bb17536be624640cdf4a

$ curl --max-time 10 -4 -vvv http://localhost:8080
* Rebuilt URL to: http://localhost:8080/
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 8080 (#0)
> GET / HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.61.1
> Accept: */*
> 
< HTTP/1.1 200 OK
< Server: nginx/1.17.8
< Date: Mon, 17 Feb 2020 18:47:55 GMT
< Content-Type: text/html
< Content-Length: 6
< Last-Modified: Mon, 17 Feb 2020 18:47:38 GMT
< Connection: keep-alive
< ETag: "5e4adfca-6"
< Accept-Ranges: bytes
< 
hello
* Connection #0 to host localhost left intact
```

Comment 40 Peter Hunt 2020-03-04 14:19:02 UTC
sorry for the ~incredibly~ late answer

In short, I'm not sure I'm a fan of changing cri-o's config files here. That would break non-openshift cases, or branch the rpm between openshift and non-openshift.

Would it be possible to change the default podman network to podman0 in the release branch for 4.2? I would make the change in cri-o, but we actually don't filter based on cni name (it was a TODO a while ago that fell to the wayside).

Comment 41 Matthew Heon 2020-03-05 13:11:26 UTC
We've already renamed the bridge interface for Podman, but the real issue is a conflict of IP address allocations. Podman and CRI-O use different bridges which both request the same subnet, which doesn't work.

I can't change this on the Podman side without breaking existing containers that set static IP Addresses.

Comment 42 Matthew Heon 2020-05-27 17:28:47 UTC
Seems like this is already fixed per comment 39.


Note You need to log in before you can comment on or make changes to this bug.