Description of problem: Run quadlet demo https://github.com/ygalblum/quadlet-demo on a recent RHEL 8.9 machine. podman ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 5a31d34f5887 docker.io/library/mysql:5.6 mysqld 20 minutes ago Up 20 minutes quadlet-demo-mysql daf8dbdbbe20 localhost/podman-pause:4.5.1-1686827770 20 minutes ago Up 20 minutes 39b3e27cf20d-service 5317ca8b8932 localhost/podman-pause:4.5.1-1686827770 20 minutes ago Up 20 minutes 0.0.0.0:8000->8080/tcp, 0.0.0.0:9000->9901/tcp e195bcda4d27-infra e06fbad62596 docker.io/library/wordpress:4.8-apache apache2-foregroun... 20 minutes ago Up 24 seconds 0.0.0.0:8000->8080/tcp, 0.0.0.0:9000->9901/tcp quadlet-demo-wordpress 083b6b99bfb9 docker.io/envoyproxy/envoy:v1.25.0 envoy -c /etc/env... 20 minutes ago Up 20 minutes 0.0.0.0:8000->8080/tcp, 0.0.0.0:9000->9901/tcp quadlet-demo-envoy you can see that the wordpress container is restarting quite often podman logs e06fbad62596 MySQL Connection Error: (2002) php_network_getaddresses: getaddrinfo failed: Name or service not known Warning: mysqli::mysqli(): php_network_getaddresses: getaddrinfo failed: Name or service not known in - on line 22 Warning: mysqli::mysqli(): (HY000/2002): php_network_getaddresses: getaddrinfo failed: Name or service not known in - on line 22 This does not happen on EL9 with podman 4.5.1, nor does it happen with the latest builds from the copr rhcontainer for epel 8 and epel 9. There must be something about podman-4.5.1-5.module+el8.9.0+19106+3ac0000c.x86_64 that is causing this problem. Version-Release number of selected component (if applicable): podman-4.5.1-5.module+el8.9.0+19106+3ac0000c.x86_64
@vrothber @yblum fyi - sorry, I don't know of a simpler way to reproduce this - but there is something definitely different about podman-4.5.1-5.module+el8.9.0+19106+3ac0000c.x86_64 that isn't in EL9 or the latest copr rhcontainer builds.
Assigning to @yblum to investigate along with @vrothber
@Jindrich, do you know of issues with this specific build?
This doesn't seem like a Quadlet issue. Having said that, where can I get the image for RHEL 8.9?
There is nothing special about this build Valentin. The only thing that comes to my mind is that it has been built with golang-1.20.4-1.module+el8.9.0+18975+0dcd002e
(In reply to Ygal Blum from comment #4) > This doesn't seem like a Quadlet issue. Having said that, where can I get > the image for RHEL 8.9? This is what I'm using for testing: Latest compose - http://download.devel.redhat.com/rhel-8/nightly/RHEL-8/latest-RHEL-8.9/compose images - http://download.devel.redhat.com/rhel-8/nightly/RHEL-8/latest-RHEL-8.9/compose/BaseOS/x86_64/images/ then you'll need to add the dnf repos: rhel-baseos http://download.devel.redhat.com/rhel-8/nightly/RHEL-8/latest-RHEL-8.9/compose/BaseOS/x86_64/os/ rhel-appstream http://download.devel.redhat.com/rhel-8/nightly/RHEL-8/latest-RHEL-8.9/compose/AppStream/x86_64/os/ I don't know if this is a quadlet issue (as opposed an issue with podman unrelated to quadlet? or one of the podman components/dependencies?) All I know is that the quadlet demo works with the podman in the latest rhel 9, fedora, and in the rhcontainer copr, and doesn't work with the podman in the latest rhel 8.9
As I suspected, I was able to reproduce this issue without Quadlet. This seems to be a DNS issue. Steps to reproduce: ------------------- 1. Create a Podman Network: podman network create repro --subnet 192.168.31.0/24 --gateway 192.168.31.1 2. Run a container in the background to be the target of nslookup: podman run --rm -d -t --network repro --name repro-target fedora:38 3. Run a container in the foreground to look for the previously created one: podman run --rm -it --network repro --name repro-origin fedora:38 3.1 In the new container installed bind-utils: dnf install -y bind-utils 3.2 Look for the target container address: nslookup repro-target Expected behavior (The actual address may vary) ----------------------------------------------- Server: 192.168.31.1 Address: 192.168.31.1#53 Non-authoritative answer: Name: repro-target.dns.podman Address: 192.168.31.4 Actual behavior --------------- Server: 192.168.122.1 Address: 192.168.122.1#53 ** server can't find repro-target: NXDOMAIN Probable root cause ------------------- I think the issue is in `/etc/resolv.conf`. Working case ------------ ``` search dns.podman nameserver 192.168.31.1 ``` Failed case ----------- ``` search 9 nameserver 192.168.122.1 ``` In the failed case, the container's resolv.conf file is equal to that of the host machine @vrothber FYI
Pulling in the network SME, Paul. Paul, do you know what's going on?
Please check `podman info --format {{.Host.NetworkBackend}}`, RHEL 8 still defaults to the old CNI stack while 9 and newer should default to netavark. CNI name resolution for containers works but you have to install podman-plugins then recreated the network config. If you run podman network inspect it should show "dns_enabled": true. If that is already installed then this is likely a podman bug where we set the incorrect resolver.
Yes, I can confirm that installing `podman-plugins` before the steps I provided solves the issue
@yblum Just checking, "solves the issue", does that pertain to the entire problem reported in here, or just your quick test with DNS? I'm just wondering if this might be a Podman packaging issue.
(In reply to Ygal Blum from comment #10) > Yes, I can confirm that installing `podman-plugins` before the steps I > provided solves the issue Should I add the package podman-plugins to https://github.com/linux-system-roles/podman/blob/main/vars/main.yml#L6? Can I add it for all platforms, or only RHEL8/Centos8?
@tsweeney , I only checked the simple test case. But, I know that root cause of the original issue is DNS resolution. Plus, AFAIK @rmeggins has tested the original use-case with `podman-plugins` and it did solve the issue.
*** This bug has been marked as a duplicate of bug 2220931 ***