https://github.com/containers/libpod/pull/3755 is working on this.
*** Bug 1747429 has been marked as a duplicate of this bug. ***
Any reason why all the discussion here was private? We'd like to link to the discussion in that bug in upstream code, but this bug is useless to !rh employees. Can we make all these comments public?
We may wish to scrub the debug logs, but I don't think anything else would be sensitive. In a brief summary, the issue is cause by conflicts between the Podman and CRI-O default CNI networks - the CRI-O network cannot successfully configure Podman networks. Normally this is not a problem, as Podman explicitly requests the Podman network be used for its containers; however, Podman 1.4.x appears to have a bug where this requested default is ignored, and the first network located (the CRI-O network) is used. Podman 1.5.x appears to have fixed the issue, but Openshift 4.2 and RHEL 8.1 are shipping Podman 1.4.2. As a workaround, removing the two CRI-O CNI networks in /etc/cni/net.d/ (100-crio-bridge.conf and 200-loopback.conf) should work fine for Openshift nodes - CRI-O networking is configured elsewhere, so the default networks are not used.
@Matthew, mcambria Is this bug is fixed with latest version of podman which now available in 4.3 release? @Micah Any testing done against 4.3 side, we are still using the workaround which @Matthew suggested https://github.com/code-ready/snc/blob/master/createdisk.sh#L341-L345 and now facing a different issue https://bugzilla.redhat.com/show_bug.cgi?id=1803635, I hope this workaround is not related to this issue.
It may be. I'll tag in Peter Hunt from the CRI-O team, as I believe the resolution (removing the unused, conflicting config files) was in their packaging.
@Praveen Testing on RHCOS 4.4 with default settings shows that we are not encountering this problem any more. ``` $ rpm-ostree status State: idle AutomaticUpdates: disabled Deployments: * ostree://a6ebea6e29826f7f619b0ecde7dd5c4c3ec5dc23c1d73393cfcf3a7e958478bd Version: 44.81.202002141514-0 (2020-02-14T15:20:58Z) $ rpm -q podman cri-o podman-1.6.4-2.module+el8.1.1+5363+bf8ff1af.x86_64 cri-o-1.17.0-4.dev.rhaos4.4.gitc3436cc.el8.x86_64 $ sudo podman run -d -p 8080:80 -v /var/tmp/index.html:/usr/share/nginx/html/index.html:z docker.io/library/nginx 19c86517c3ff781d29ffcf9ed8fe8d1b51598557b80629a9fb087971e6468727 [core@coreos ~]$ curl --max-time 10 -4 -vvv http://localhost:8080 * Rebuilt URL to: http://localhost:8080/ * Trying 127.0.0.1... * TCP_NODELAY set * Connected to localhost (127.0.0.1) port 8080 (#0) > GET / HTTP/1.1 > Host: localhost:8080 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Server: nginx/1.17.8 < Date: Mon, 17 Feb 2020 18:45:46 GMT < Content-Type: text/html < Content-Length: 6 < Last-Modified: Mon, 17 Feb 2020 18:44:00 GMT < Connection: keep-alive < ETag: "5e4adef0-6" < Accept-Ranges: bytes < hello * Connection #0 to host localhost left intact ``` Additionally, RHCOS 4.3 with default settings is also successful: ``` $ rpm-ostree status rState: idle AutomaticUpdates: disabled Deployments: * ostree://08ee6458b08e6e0a5b773ecd189cf842b58d685ff8eda88d241a7ad54f22aad2 Version: 43.81.202002110953.0 (2020-02-11T09:59:02Z) $ rpm -q cri-o podman cri-o-1.16.3-19.dev.rhaos4.3.git6c1f4bd.el8.x86_64 podman-1.6.4-2.module+el8.1.1+5363+bf8ff1af.x86_64 $ sudo podman run -d -p 8080:80 -v /var/tmp/index.html:/usr/share/nginx/html/index.html:z docker.io/library/nginx Trying to pull docker.io/library/nginx... Getting image source signatures Copying blob bf317aa10aa5 done Copying blob bc51dd8edc1b done Copying blob 66ba67045f57 done Copying config 2073e0bcb6 done Writing manifest to image destination Storing signatures 7a2d31d699b67d89223a48c81ec29ee54884d10a3e44bb17536be624640cdf4a $ curl --max-time 10 -4 -vvv http://localhost:8080 * Rebuilt URL to: http://localhost:8080/ * Trying 127.0.0.1... * TCP_NODELAY set * Connected to localhost (127.0.0.1) port 8080 (#0) > GET / HTTP/1.1 > Host: localhost:8080 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Server: nginx/1.17.8 < Date: Mon, 17 Feb 2020 18:47:55 GMT < Content-Type: text/html < Content-Length: 6 < Last-Modified: Mon, 17 Feb 2020 18:47:38 GMT < Connection: keep-alive < ETag: "5e4adfca-6" < Accept-Ranges: bytes < hello * Connection #0 to host localhost left intact ```
sorry for the ~incredibly~ late answer In short, I'm not sure I'm a fan of changing cri-o's config files here. That would break non-openshift cases, or branch the rpm between openshift and non-openshift. Would it be possible to change the default podman network to podman0 in the release branch for 4.2? I would make the change in cri-o, but we actually don't filter based on cni name (it was a TODO a while ago that fell to the wayside).
We've already renamed the bridge interface for Podman, but the real issue is a conflict of IP address allocations. Podman and CRI-O use different bridges which both request the same subnet, which doesn't work. I can't change this on the Podman side without breaking existing containers that set static IP Addresses.
Seems like this is already fixed per comment 39.