Bug 2019901

Summary: ubi9-beta/ubi-minimal has a broken microdnf (g_system_thread_new fatal error)
Product: Red Hat Enterprise Linux 8 Reporter: Jonathan Dowland <jdowland>
Component: containers-commonAssignee: Jindrich Novy <jnovy>
Status: CLOSED CURRENTRELEASE QA Contact: Yuhui Jiang <yujiang>
Severity: medium Docs Contact: Gabriela Nečasová <gnecasov>
Priority: urgent    
Version: 8.4CC: bagasse, dornelas, fweimer, gnecasov, gscrivan, jnovy, jvmartin, jwboyer, lmanasko, mheon, pasik, toneata, tsweeney, walters, ypu, yujiang
Target Milestone: rcKeywords: Triaged, ZStream
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
.UBI 9-Beta containers can run on RHEL 7 and 8 hosts Previously, the UBI 9-Beta container images had an incorrect seccomp profile set in the `containers-common` package. As a consequence, containers were not able to deal with certain system calls causing a failure. With this update, the problem has been fixed.
Story Points: ---
Clone Of:
: 2095807 2095808 2095821 (view as bug list) Environment:
Last Closed: 2023-11-30 15:11:01 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 2095807, 2095808    

Description Jonathan Dowland 2021-11-03 15:22:57 UTC
Description of problem:

microdnf in the ubi9-beta/ubi-minimal image appears to be broken.

Version-Release number of selected component (if applicable):

container version 9.0.0beta-319
microdnf-3.8.0-3.el9.x86_64


How reproducible:

always

Steps to Reproduce:
1. run microdnf
2.
3.

Actual results:

(microdnf:1): GLib-ERROR **: 15:21:00.016: file ../glib/gthread-posix.c: line 1338 (g_system_thread_new): error 'Operation not permitted' during 'pthread_create'

Expected results:

install/remove packages etc

Additional info:

Comment 1 Colin Walters 2021-11-04 01:02:39 UTC
Are you using docker?  If so, then this is a dup of https://github.com/moby/moby/pull/42681

Comment 2 Jonathan Dowland 2021-11-04 09:29:15 UTC
I've reproduced this using docker (version 20.10.5+dfsg1-1+b5 from Debian 11 fwiw) but also podman-3.2.3-0.11.module+el8.4.0+12050+ef972f71.x86_64 on RHEL 8.4 for the host.

Comment 3 Josh Boyer 2021-11-04 12:57:25 UTC
RHEL 8.4 podman doesn't have a seccomp profile that works with the newer clone3 syscall used in RHEL 9 binaries.  For now, you need to run it with --security-opt seccomp=unconfined.  This will be addressed in a future RHEL 8 release.

[jwboyer@vader ~]$ podman run -it --rm registry.access.redhat.com/ubi9-beta/ubi-minimal
Trying to pull registry.access.redhat.com/ubi9-beta/ubi-minimal:latest...
Getting image source signatures
Checking if image destination supports signatures
Copying blob de61ad23d5af done  
Copying blob 34e48c21a445 done  
Copying config fcf4fa2ad9 done  
Writing manifest to image destination
Storing signatures
[root@e772e5d6a752 /]# microdnf

(microdnf:12): GLib-ERROR **: 12:55:48.732: file ../glib/gthread-posix.c: line 1338 (g_system_thread_new): error 'Operation not permitted' during 'pthread_create'
Trace/breakpoint trap (core dumped)
[root@e772e5d6a752 /]# exit
exit
[jwboyer@vader ~]$ podman run -it --rm --security-opt seccomp=unconfined registry.access.redhat.com/ubi9-beta/ubi-minimal
[root@654b8a0698cd /]# microdnf 
This is microdnf, which implements subset of `dnf'.
Usage:
  microdnf [OPTION…] COMMAND

Commands:
  upgrade              Upgrade packages
  update               Compatibility alias for the "upgrade" command
  module reset         Reset a module stream
  distro-sync          Upgrade/downgrade packages to match versions in repositories
  dsync                Compatibility alias for the "distro-sync" command
  remove               Remove packages
  reinstall            Reinstall packages
  clean                Remove cached data
  repolist             List repositories
  module enable        Enable a module stream
  download             Download packages
  makecache            Generate the metadata cache
  module disable       Disable a module stream
  repoquery            Search for packages matching keyword
  install              Install packages

Help Options:
  -h, --help                    Show help options

Application Options:
  --assumeno                    Automatically answer no for all questions
  -y, --assumeyes               Automatically answer yes for all questions
  --best                        Try the best available package versions in transactions
  --config=<config file>        Configuration file location
  --disablerepo=ID              Disable repository by an id
  --disableplugin=name          Disable plugins by name
  --enablerepo=ID               Enable repository by an id
  --enableplugin=name           Enable plugins by name
  --nobest                      Do not limit the transaction to the best candidates
  --installroot=PATH            Set install root
  --nodocs                      Install packages without docs
  --noplugins                   Disable loading of plugins
  --refresh                     Set metadata as expired before running the command
  --releasever=RELEASEVER       Override the value of $releasever in config and repo files
  --setopt=<option>=<value>     Override a configuration option (install_weak_deps=0/1, allow_vendor_change=0/1, keepcache=0/1, module_platform_id=<name:stream>, cachedir=<path>, reposdir=<path1>,<path2>,..., tsflags=nodocs/test, varsdir=<path1>,<path2>,..., repo_id.option_name=<value>)

error: No command specified
[root@654b8a0698cd /]#

Comment 4 Scott McCarty 2021-11-09 07:25:33 UTC
This appears to affect more than just te clone() syscall. I verified this on a RHEL 7 and RHEL 8 host using the RHEL 9 container image:

[root@rhel7 ~]# podman run -it registry.access.redhat.com/ubi9-beta/ubi bash
[root@e094e2ec5677 /]# curl www.redhat.com
curl: (6) getaddrinfo() thread failed to start



But, this works fine:
[root@rhel7 ~]# podman run -it --security-opt seccomp=unconfined registry.access.redhat.com/ubi9-beta/ubi bash
[root@3416996cb2fe /]# curl www.redhat.com
[root@3416996cb2fe /]#

Comment 5 Giuseppe Scrivano 2021-11-16 14:44:04 UTC
could you please check how your seccomp profile looks like?

A  seccomp profile, that deals correctly with new syscalls, looks like the following:

$ head -3 /usr/share/containers/seccomp.json
{
	"defaultAction": "SCMP_ACT_ERRNO",
	"defaultErrnoRet": 38,

If there is no defaultErrnoRet, then the runtime uses by default EPERM causing glibc to fail.

Comment 6 Matthew Heon 2021-11-16 20:12:40 UTC
After investigation, the Seccomp profile in RHEL's containers/common is not correct; the latest upstream version will return ENOSYS and not EPERM, which will cause libc in the container to assume (correctly) the kernel does not have support for the new syscall and fall back to older syscalls. We'll need this in RHEL 8.x and 9.0 both.

Comment 7 Derrick Ornelas 2021-11-17 15:00:26 UTC
containers-common-1-2.module+el8.5.0+12582+56d94c81 shipped with RHEL 8.5 contains the following seccomp.json config:

# rpm -qf /usr/share/containers/seccomp.json
containers-common-1-2.module+el8.5.0+12582+56d94c81.noarch


# head -n5  /usr/share/containers/seccomp.json 
{
	"defaultAction": "SCMP_ACT_ERRNO",
	"defaultErrnoRet": 38,
	"archMap": [
		{



So I think this is fixed in RHEL 8 already.  Users just need to update.  Someone will need to check RHEL 9 containers-common.

Comment 9 Derrick Ornelas 2021-11-17 15:15:27 UTC
RHEL 9 Beta ships containers-common-1-2.module+el9.0.0+12467+378c8264 which ships this seccomp.json:  

# rpm2cpio ../containers-common-1-2.module+el9.0.0+12467+378c8264.noarch.rpm | cpio -id
480 blocks

# head -n5 ./usr/share/containers/seccomp.json 
{
	"defaultAction": "SCMP_ACT_ERRNO",
	"defaultErrnoRet": 38,
	"archMap": [
		{

Comment 10 Tom Sweeney 2021-11-17 23:02:28 UTC
@jnovy can you verify the containers-common that's we're pointing at RHEL 9 and 8.6 when you get a chance please?

Comment 11 Gabriela Nečasová 2021-11-22 08:32:51 UTC
Derrick or Tom, 
I tried to write some docText. Can you check it please? 
Many thanks.

Comment 12 Jindrich Novy 2021-11-22 10:48:30 UTC
The seccomp.json rules were fixed in the way that the profile is not delivered from the main upstream branch of c/common but the oldest vendored version of c/common in podman, skopeo and buildah:

http://pkgs.devel.redhat.com/cgit/rpms/containers-common/commit/?h=stream-container-tools-rhel8-rhel-8.6.0&id=d0c5f615fa52a6edcc8e8356cfd0e2113c63e433
http://pkgs.devel.redhat.com/cgit/rpms/containers-common/commit/?h=rhel-9.0.0&id=d9fd99ed3d5e1b732b8c0db2e73ee61373ea30d4

Comment 24 J. Victor Martins 2022-05-19 18:14:14 UTC
I can reproduce this issue when building UBI9-based images in CircleCI docker runtime, the output is:

```
Step 7/28 : RUN microdnf -y install tar gzip && tar -zxf /bundle.tar.gz
 ---> Running in 13ccd336fe4f

(microdnf:8): GLib-ERROR **: 23:33:11.649: file ../glib/gthread-posix.c: line 1338 (g_system_thread_new): error 'Operation not permitted' during 'pthread_create'
The command '/bin/sh -c microdnf -y install tar gzip && tar -zxf /bundle.tar.gz' returned a non-zero code: 133
make: *** [Makefile:187: scanner-image] Error 133
```

The build runtime information:

```
Build-agent version 1.0.123804-6379e05c (2022-05-18T21:49:50+0000)
System information:
 Server Version: 20.10.15
 Storage Driver: overlay2
  Backing Filesystem: xfs
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Kernel Version: 5.13.0-1021-aws
 Operating System: Ubuntu 20.04.4 LTS
 OSType: linux
 Architecture: x86_64
```