Bug 1990469

Summary: docker rhel9 vm not resolving with "getaddrinfo() thread failed to start"
Product: Red Hat Enterprise Linux 7 Reporter: michal novacek <mnovacek>
Component: dockerAssignee: Daniel Walsh <dwalsh>
Status: CLOSED WONTFIX QA Contact: Alex Jia <ajia>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 7.6CC: ajia, amurdaca, jnovy, lsm5, pasik, redhat.com, smccarty, tsweeney, walters, ypu
Target Milestone: rcKeywords: Extras
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-12-07 12:28:07 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description michal novacek 2021-08-05 13:17:51 UTC
Description of problem:
Starting rhel9 docker container in rhel7.6 (vm) will resolve in container that is unable to resolve with "getaddrinfo() thread failed to start".


Version-Release number of selected component (if applicable):
rhel7.6
docker-1.13.1-101.git61c8524.el7.x86_64
kernel-3.10.0-957.10.1.el7.x86_64

How reproducible: always


Steps to Reproduce:
1. rhel7.6-vm# docker run -ti --rm --hostname test registry-proxy.engineering.redhat.com/rh-osbs/ubi9:9.0.0beta-476 /bin/bash
2. container# curl http://www.redhat.com

Actual results: getaddrinfo() thread failed to start", no rosolving

Expected results: address resolved

Additional info:

gitlab-runner-1 is rhel7.6 vm.

> [root@gitlab-runner-1 ~]# host www.redhat.com
www.redhat.com is an alias for ds-www.redhat.com.edgekey.net.
ds-www.redhat.com.edgekey.net is an alias for ds-www.redhat.com.edgekey.net.globalredir.akadns.net.
ds-www.redhat.com.edgekey.net.globalredir.akadns.net is an alias for e3396.dscx.akamaiedge.net.
e3396.dscx.akamaiedge.net has address 104.73.38.122
e3396.dscx.akamaiedge.net has IPv6 address 2a02:26f0:fe00:190::d44
e3396.dscx.akamaiedge.net has IPv6 address 2a02:26f0:fe00:1a5::d44

> [root@gitlab-runner-1 ~]# cat /etc/resolv.conf
# Generated by NetworkManager
search cluster-qe.lab.eng.brq.redhat.com
nameserver 10.37.166.1

> [root@gitlab-runner-1 ~]# docker run -ti --rm --hostname test registry-proxy.engineering.redhat.com/rh-osbs/ubi9:9.0.0beta-476 /bin/bash

> [root@test /]# cat /etc/redhat-release
Red Hat Enterprise Linux release 9.0 Beta (Plow)

>[root@test /]# cat /etc/resolv.conf
# Generated by NetworkManager
search cluster-qe.lab.eng.brq.redhat.com
nameserver 10.37.166.1

> [root@test /]# curl http://www.redhat.com
curl: (6) getaddrinfo() thread failed to start

Comment 3 michal novacek 2021-08-05 15:16:25 UTC
The same behaviour can be observed on physical hardware with all packages upgraded to latest Z-stream.

Package versions:

kernel-3.10.0-957.61.1.el7.x86_64
docker-1.13.1-102.git7f2769b.el7.x86_64


Running ubi8 image instead of ubi9 the container can resolve correctly.

Comment 5 Colin Walters 2021-08-05 17:50:09 UTC
This is just the latest iteration of seccomp rules vs glibc, see e.g. https://bugzilla.redhat.com/show_bug.cgi?id=1900021 and https://bugzilla.redhat.com/show_bug.cgi?id=1985499

We need to remind our customers of this matrix https://access.redhat.com/support/policy/rhel-container-compatibility (which should be extended to include RHEL9) and that the general rule is that you can run RHEL N on a RHEL N+1 kernel, but not RHEL N+1 on RHEL N kernel.  And in this case, 7 is much less than 9.

Basically if you want to run RHEL9 or modern userspace in a container, keep your host up to date.

Comment 6 Colin Walters 2021-08-05 17:51:44 UTC
It's certainly be really helpful for glibc if we backported the "return ENOSYS instead of EPERM" bits to our docker/podman as far back as is reasonable though.  But OTOH it's exactly the kind of churn that also incurs some risk in the RHEL7 lifecycle now.

Comment 7 michal novacek 2021-08-06 12:10:30 UTC
(In reply to Colin Walters from comment #5)
> This is just the latest iteration of seccomp rules vs glibc, see e.g.
> https://bugzilla.redhat.com/show_bug.cgi?id=1900021 and
> https://bugzilla.redhat.com/show_bug.cgi?id=1985499
> 
> We need to remind our customers of this matrix
> https://access.redhat.com/support/policy/rhel-container-compatibility (which
> should be extended to include RHEL9) and that the general rule is that you
> can run RHEL N on a RHEL N+1 kernel, but not RHEL N+1 on RHEL N kernel.  And
> in this case, 7 is much less than 9.
> 
> Basically if you want to run RHEL9 or modern userspace in a container, keep
> your host up to date.


I understand that having rhel7.6 as a host is way behind so what would be recommended solution to run rhel9 containers? 

We cannot really run rhel9 for production purposes so my expectation would be that rhel8 should be able to run rhel9 containers. Is it?

Comment 8 Tom Sweeney 2021-08-06 15:21:35 UTC
Michal,

Yes, you can run rhel9 containers on a rhel8 host.  There has been some early testing for that.  Just a note, you would be running using Podman instead of Docker.

Comment 9 John Villalovos 2021-08-25 16:39:41 UTC
I ran into this issue on Ubuntu 20.04 when trying to run a Fedora rawhide image.

The work around was to use: --security-opt seccomp=unconfined
on the docker command line.

I am also trying to run the image with a Gitlab runner but the OS is Ubuntu 20.04.

Comment 10 Scott McCarty 2022-01-19 16:06:15 UTC
John, could you highlight which way is which, do you mean you are running an Ubuntu container image on a Fedora rawhide host, or vice versa?

Comment 11 John Villalovos 2022-01-19 17:35:33 UTC
(In reply to Scott McCarty from comment #10)
> John, could you highlight which way is which, do you mean you are running an
> Ubuntu container image on a Fedora rawhide host, or vice versa?

Ubuntu 20.04 host and running Fedora as the container. From my understanding this is because of change in Fedora on the version of glibc.

Comment 12 Scott McCarty 2022-12-06 01:14:18 UTC
All, did this ever get resolved?

Comment 14 Scott McCarty 2022-12-07 12:28:07 UTC
All, I'm closing this issue as running a RHEL9 container image on a RHEL7 host is not supported per: https://access.redhat.com/support/policy/rhel-container-compatibility