Bug 2218917

Summary: quadlet demo wordpress container fails with php_network_getaddresses: getaddrinfo failed: Name or service not known in - on line 22
Product: Red Hat Enterprise Linux 8 Reporter: Rich Megginson <rmeggins>
Component: podmanAssignee: Ygal Blum <yblum>
Status: CLOSED DUPLICATE QA Contact: atomic-bugs <atomic-bugs>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 8.9CC: bbaude, dwalsh, jligon, jnovy, lsm5, mboddu, mheon, pholzing, pthomas, tsweeney, umohnani, vrothber, yblum
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-07-06 14:47:59 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Rich Megginson 2023-06-30 14:38:50 UTC
Description of problem:
Run quadlet demo https://github.com/ygalblum/quadlet-demo on a recent RHEL 8.9 machine.

podman ps -a

CONTAINER ID  IMAGE                                    COMMAND               CREATED         STATUS         PORTS                                           NAMES
5a31d34f5887  docker.io/library/mysql:5.6              mysqld                20 minutes ago  Up 20 minutes                                                  quadlet-demo-mysql
daf8dbdbbe20  localhost/podman-pause:4.5.1-1686827770                        20 minutes ago  Up 20 minutes                                                  39b3e27cf20d-service
5317ca8b8932  localhost/podman-pause:4.5.1-1686827770                        20 minutes ago  Up 20 minutes  0.0.0.0:8000->8080/tcp, 0.0.0.0:9000->9901/tcp  e195bcda4d27-infra
e06fbad62596  docker.io/library/wordpress:4.8-apache   apache2-foregroun...  20 minutes ago  Up 24 seconds  0.0.0.0:8000->8080/tcp, 0.0.0.0:9000->9901/tcp  quadlet-demo-wordpress
083b6b99bfb9  docker.io/envoyproxy/envoy:v1.25.0       envoy -c /etc/env...  20 minutes ago  Up 20 minutes  0.0.0.0:8000->8080/tcp, 0.0.0.0:9000->9901/tcp  quadlet-demo-envoy

you can see that the wordpress container is restarting quite often

podman logs e06fbad62596

MySQL Connection Error: (2002) php_network_getaddresses: getaddrinfo failed: Name or service not known

Warning: mysqli::mysqli(): php_network_getaddresses: getaddrinfo failed: Name or service not known in - on line 22

Warning: mysqli::mysqli(): (HY000/2002): php_network_getaddresses: getaddrinfo failed: Name or service not known in - on line 22

This does not happen on EL9 with podman 4.5.1, nor does it happen with the latest builds from the copr rhcontainer for epel 8 and epel 9.  There must be something about podman-4.5.1-5.module+el8.9.0+19106+3ac0000c.x86_64 that is causing this problem.

Version-Release number of selected component (if applicable):

podman-4.5.1-5.module+el8.9.0+19106+3ac0000c.x86_64

Comment 1 Rich Megginson 2023-06-30 14:42:33 UTC
@vrothber @yblum fyi - sorry, I don't know of a simpler way to reproduce this - but there is something definitely different about podman-4.5.1-5.module+el8.9.0+19106+3ac0000c.x86_64 that isn't in EL9 or the latest copr rhcontainer builds.

Comment 2 Tom Sweeney 2023-06-30 17:51:16 UTC
Assigning to @yblum to investigate along with @vrothber

Comment 3 Valentin Rothberg 2023-07-03 06:17:30 UTC
@Jindrich, do you know of issues with this specific build?

Comment 4 Ygal Blum 2023-07-03 06:21:26 UTC
This doesn't seem like a Quadlet issue. Having said that, where can I get the image for RHEL 8.9?

Comment 5 Jindrich Novy 2023-07-03 08:03:53 UTC
There is nothing special about this build Valentin. The only thing that comes to my mind is that it has been built with golang-1.20.4-1.module+el8.9.0+18975+0dcd002e

Comment 6 Rich Megginson 2023-07-03 18:16:34 UTC
(In reply to Ygal Blum from comment #4)
> This doesn't seem like a Quadlet issue. Having said that, where can I get
> the image for RHEL 8.9?

This is what I'm using for testing:

Latest compose - http://download.devel.redhat.com/rhel-8/nightly/RHEL-8/latest-RHEL-8.9/compose
images - http://download.devel.redhat.com/rhel-8/nightly/RHEL-8/latest-RHEL-8.9/compose/BaseOS/x86_64/images/
then you'll need to add the dnf repos:
rhel-baseos http://download.devel.redhat.com/rhel-8/nightly/RHEL-8/latest-RHEL-8.9/compose/BaseOS/x86_64/os/
rhel-appstream http://download.devel.redhat.com/rhel-8/nightly/RHEL-8/latest-RHEL-8.9/compose/AppStream/x86_64/os/

I don't know if this is a quadlet issue (as opposed an issue with podman unrelated to quadlet?  or one of the podman components/dependencies?)

All I know is that the quadlet demo works with the podman in the latest rhel 9, fedora, and in the rhcontainer copr, and doesn't work with the podman in the latest rhel 8.9

Comment 7 Ygal Blum 2023-07-04 10:14:43 UTC
As I suspected, I was able to reproduce this issue without Quadlet. This seems to be a DNS issue.

Steps to reproduce:
-------------------
1. Create a Podman Network:
podman network create repro --subnet 192.168.31.0/24 --gateway 192.168.31.1
2. Run a container in the background to be the target of nslookup:
podman run --rm -d -t --network repro --name repro-target fedora:38
3. Run a container in the foreground to look for the previously created one:
podman run --rm -it --network repro --name repro-origin fedora:38
3.1 In the new container installed bind-utils:
dnf install -y bind-utils
3.2 Look for the target container address:
nslookup repro-target

Expected behavior (The actual address may vary)
-----------------------------------------------
Server:         192.168.31.1
Address:        192.168.31.1#53

Non-authoritative answer:
Name:   repro-target.dns.podman
Address: 192.168.31.4

Actual behavior
---------------
Server:         192.168.122.1
Address:        192.168.122.1#53

** server can't find repro-target: NXDOMAIN

Probable root cause
-------------------
I think the issue is in `/etc/resolv.conf`. 

Working case
------------
```
search dns.podman
nameserver 192.168.31.1
```

Failed case
-----------
```
search 9
nameserver 192.168.122.1
```

In the failed case, the container's resolv.conf file is equal to that of the host machine

@vrothber FYI

Comment 8 Valentin Rothberg 2023-07-04 11:01:59 UTC
Pulling in the network SME, Paul.  Paul, do you know what's going on?

Comment 9 Paul Holzinger 2023-07-04 11:18:25 UTC
Please check `podman info --format {{.Host.NetworkBackend}}`, RHEL 8 still defaults to the old CNI stack while 9 and newer should default to netavark.

CNI name resolution for containers works but you have to install podman-plugins then recreated the network config. If you run podman network inspect it should show "dns_enabled": true.

If that is already installed then this is likely a podman bug where we set the incorrect resolver.

Comment 10 Ygal Blum 2023-07-04 11:50:23 UTC
Yes, I can confirm that installing `podman-plugins` before the steps I provided solves the issue

Comment 11 Tom Sweeney 2023-07-05 15:31:35 UTC
@yblum Just checking, "solves the issue", does that pertain to the entire problem reported in here, or just your quick test with DNS?  I'm just wondering if this might be a Podman packaging issue.

Comment 12 Rich Megginson 2023-07-05 19:08:55 UTC
(In reply to Ygal Blum from comment #10)
> Yes, I can confirm that installing `podman-plugins` before the steps I
> provided solves the issue

Should I add the package podman-plugins to https://github.com/linux-system-roles/podman/blob/main/vars/main.yml#L6?  Can I add it for all platforms, or only RHEL8/Centos8?

Comment 13 Ygal Blum 2023-07-06 06:34:58 UTC
@tsweeney , I only checked the simple test case. But, I know that root cause of the original issue is DNS resolution. Plus, AFAIK @rmeggins has tested the original use-case with `podman-plugins` and it did solve the issue.

Comment 15 Rich Megginson 2023-07-06 14:47:59 UTC

*** This bug has been marked as a duplicate of bug 2220931 ***