Bug 1965093

Summary: Upgrade of CentOS Stream container fails
Product: Red Hat Enterprise Linux 8 Reporter: Justin Jereza <justinjereza>
Component: filesystemAssignee: Pavel Zhukov <pzhukov>
Status: CLOSED DUPLICATE QA Contact: rhel-cs-infra-services-qe <rhel-cs-infra-services-qe>
Severity: medium Docs Contact:
Priority: unspecified    
Version: CentOS StreamCC: bbaude, bstinson, dwalsh, jligon, jnovy, jwboyer, lsm5, mheon, pthomas, tsweeney, umohnani
Target Milestone: beta   
Target Release: ---   
Hardware: x86_64   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-06-21 06:16:18 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Output of podman build on CentOS
none
Output of podman build on RHEL none

Description Justin Jereza 2021-05-26 19:59:04 UTC
Description of problem:

The following when built with (rootless) `podman build` will encounter a transaction failure on CentOS Stream but works fine on RHEL 8.

```
FROM quay.io/centos/centos:stream
RUN dnf --assumeyes upgrade
```

Output of dnf upgrade:

```
Installed:
  cpio-2.12-10.el8.x86_64                cracklib-dicts-2.9.6-15.el8.x86_64        diffutils-3.6-6.el8.x86_64                      
  dracut-049-136.git20210426.el8.x86_64  file-5.33-20.el8.x86_64                   gettext-0.19.8.1-17.el8.x86_64                  
  gettext-libs-0.19.8.1-17.el8.x86_64    glibc-langpack-en-2.28-158.el8.x86_64     grub2-common-1:2.02-99.el8.noarch               
  grub2-tools-1:2.02-99.el8.x86_64       grub2-tools-minimal-1:2.02-99.el8.x86_64  grubby-8.40-41.el8.x86_64                       
  hardlink-1:1.3-6.el8.x86_64            kbd-2.0.4-10.el8.x86_64                   kbd-legacy-2.0.4-10.el8.noarch                  
  kbd-misc-2.0.4-10.el8.noarch           kmod-25-17.el8.x86_64                     kpartx-0.8.4-13.el8.x86_64                      
  libcroco-0.6.12-4.el8_2.1.x86_64       libevent-2.1.8-5.el8.x86_64               libgomp-8.4.1-2.1.el8.x86_64                    
  libkcapi-1.2.0-2.el8.x86_64            libkcapi-hmaccalc-1.2.0-2.el8.x86_64      libxkbcommon-0.9.1-1.el8.x86_64                 
  memstrack-0.1.11-1.el8.x86_64          openssl-1:1.1.1g-15.el8_3.x86_64          openssl-pkcs11-0.4.10-2.el8.x86_64              
  os-prober-1.74-6.el8.x86_64            pigz-2.4-4.el8.x86_64                     platform-python-pip-9.0.3-19.el8.noarch         
  procps-ng-3.3.15-6.el8.x86_64          python3-unbound-1.7.3-17.el8.x86_64       rpm-plugin-systemd-inhibit-4.14.3-13.el8.x86_64 
  shared-mime-info-1.9-3.el8.x86_64      systemd-udev-239-45.el8.x86_64            trousers-0.3.15-1.el8.x86_64                    
  trousers-lib-0.3.15-1.el8.x86_64       xkeyboard-config-2.28-1.el8.noarch        xz-5.2.4-3.el8.x86_64                           
Failed:
  unbound-libs-1.7.3-17.el8.x86_64                                                                                                 

Error: Transaction failed
```

dnf.log has a stack trace.

```
2021-05-26T19:36:36+0000 DDEBUG /var/cache/dnf/baseos-055ffcb2ec25a27f/packages/libdnf-0.55.0-7.el8.x86_64.rpm removed
2021-05-26T19:36:36+0000 SUBDEBUG 
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/dnf/cli/main.py", line 67, in main
    return _main(base, args, cli_class, option_parser_class)
  File "/usr/lib/python3.6/site-packages/dnf/cli/main.py", line 106, in _main
    return cli_run(cli, base)
  File "/usr/lib/python3.6/site-packages/dnf/cli/main.py", line 130, in cli_run
    ret = resolving(cli, base)
  File "/usr/lib/python3.6/site-packages/dnf/cli/main.py", line 176, in resolving
    base.do_transaction(display=displays)
  File "/usr/lib/python3.6/site-packages/dnf/cli/cli.py", line 262, in do_transaction
    if tsi.state == libdnf.transaction.TransactionItemState_ERROR:
dnf.exceptions.Error: Transaction failed
2021-05-26T19:36:36+0000 CRITICAL Error: Transaction failed
```

Version-Release number of selected component (if applicable):

podman-3.1.0-0.13.module_el8.5.0+733+9bb5dffa.x86_64

How reproducible:

Always

Steps to Reproduce:

`podman build` an image with the Containerfile mentioned above.

Actual results:

Image creation failure.

Expected results:

Successful image creation.

Comment 1 Tom Sweeney 2021-05-29 22:23:47 UTC
Lokesh,

Can you take an initial look at this, please?  Feel free to spin it off as necessary.

Comment 2 Daniel Walsh 2021-06-11 14:05:07 UTC
I doubt this is related to podman release.

Has anyone else seen this happen?  Is it repeatable?

Comment 3 Daniel Walsh 2021-06-11 14:05:58 UTC
Were you able to get any containers to run? And are you sure the account is setup correctly?

Comment 4 Justin Jereza 2021-06-12 16:18:53 UTC
(In reply to Daniel Walsh from comment #3)
> Were you able to get any containers to run? And are you sure the account is
> setup correctly?

What account are you referring to? If it's $USER, as I understand it there should be no special setup required. Pulling `quay.io/centos/centos:stream` also does not require authentication.

I created separate roots on both platforms just to ensure there's nothing in storage that might be affecting it and pulled a completely new image. It now fails to build on both CentOS and RHEL so you're probably right that it's not related to the podman release. The one difference though is that building on CentOS fails to install `unbound-libs-1.7.3-17.el8.x86_64` while it is not shown as a failure on RHEL. That part I'm not yet clear on where the problem lies exactly.

I have no problem building other containers like alpine.

I will attach the output of `podman build` on both platforms.

Comment 5 Justin Jereza 2021-06-12 16:19:46 UTC
Created attachment 1790495 [details]
Output of podman build on CentOS

Comment 6 Justin Jereza 2021-06-12 16:20:18 UTC
Created attachment 1790496 [details]
Output of podman build on RHEL

Comment 7 Daniel Walsh 2021-06-14 13:42:06 UTC
This looks like the filesystem package does not like being installed within a container.

```
  Upgrading        : filesystem-3.8-4.el8.0.1.x86_64                      2/203Error unpacking rpm package filesystem-3.8-4.el8.0.1.x86_64
 
  Installing       : cracklib-dicts-2.9.6-15.el8.x86_64                   3/203 
error: unpacking of archive failed on file /proc: cpio: chown failed - Inappropriate ioctl for device
error: filesystem-3.8-4.el8.0.1.x86_64: install failed

```

It looks like it is trying to do something with /proc?

Comment 8 Pavel Zhukov 2021-06-21 06:16:18 UTC

*** This bug has been marked as a duplicate of bug 1912155 ***