Bug 2183034 - packages with file capabilities fail to install in podman on certain upgraded hosts
Summary: packages with file capabilities fail to install in podman on certain upgraded...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: fuse3
Version: 38
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Peter Lemenkov
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard: RejectedBlocker AcceptedFreezeException
Depends On:
Blocks: F38FinalFreezeException
TreeView+ depends on / blocked
 
Reported: 2023-03-30 09:33 UTC by Boyd
Modified: 2023-04-18 06:29 UTC (History)
44 users (show)

Fixed In Version: fuse3-3.13.1-2.fc38
Clone Of:
Environment:
Last Closed: 2023-04-13 20:09:40 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)
podman info (3.03 KB, text/plain)
2023-04-03 12:09 UTC, Kamil Páral
no flags Details

Description Boyd 2023-03-30 09:33:19 UTC
Description of problem: dnf update of fresh creation of fedora toolbox 38 fails with
Failed:
  shadow-utils-2:4.13-4.fc38.x86_64                     shadow-utils-2:4.13-6.fc38.x86_64                    


Version-Release number of selected component (if applicable):


How reproducible:




Steps to reproduce:

1. toolbox enter fedora-toolbox-38
2. su -
3. dnf update -y


Actual results:
Failed:
  shadow-utils-2:4.13-4.fc38.x86_64                     shadow-utils-2:4.13-6.fc38.x86_64        

Expected results:
No failure


Additional info:

Error unpacking rpm package shadow-utils-2:4.13-6.fc38.x86_64
  Running scriptlet: rpm-4.18.1-1.fc38.x86_64                                                                  43/116 
error: unpacking of archive failed on file /usr/bin/newgidmap;64254ae7: cpio: cap_set_file failed - No data available
error: shadow-utils-2:4.13-6.fc38.x86_64: install failed

Comment 1 Iker Pedrosa 2023-03-30 10:44:47 UTC
I'm not an expert with toolbox, but you forgot to add one command. Apart from that everything's working on my side:
1. toolbox create --release 38
2. toolbox enter fedora-toolbox-38
3. su -
4. dnf update -y
...
 shadow-utils                                                          x86_64                                     2:4.13-6.fc38                                                     fedora                                              1.3 M
...
41/116 
  Upgrading        : shadow-utils-2:4.13-6.fc38.x86_64                                                                                                                                                                                  
...

5. dnf info shadow-utils
Last metadata expiration check: 0:04:00 ago on Thu Mar 30 12:39:20 2023.
Installed Packages
Name         : shadow-utils
Epoch        : 2
Version      : 4.13
Release      : 6.fc38
Architecture : x86_64
Size         : 4.0 M
Source       : shadow-utils-4.13-6.fc38.src.rpm
Repository   : @System
From repo    : fedora
Summary      : Utilities for managing accounts and shadow password files
URL          : https://github.com/shadow-maint/shadow
License      : BSD-3-Clause AND GPL-2.0-or-later
Description  : The shadow-utils package includes the necessary programs for
             : converting UNIX password files to the shadow password format, plus
             : programs for managing user and group accounts. The pwconv command
             : converts passwords to the shadow password format. The pwunconv command
             : unconverts shadow passwords and generates a passwd file (a standard
             : UNIX password file). The pwck command checks the integrity of password
             : and shadow files. The lastlog command prints out the last login times
             : for all users. The useradd, userdel, and usermod commands are used for
             : managing user accounts. The groupadd, groupdel, and groupmod commands
             : are used for managing group accounts.


Can you try again? If you can provide more information that would be helpful.

Comment 2 Kamil Páral 2023-03-30 12:26:07 UTC
That's interesting, I can reproduce the problem just fine.

$ toolbox list
IMAGE ID      IMAGE NAME                                    CREATED
901c633cace2  registry.fedoraproject.org/fedora-toolbox:38  4 weeks ago

$ toolbox create --release 38 f38toolbox

$ toolbox enter f38toolbox

$ sudo dnf update shadow-utils
Last metadata expiration check: 0:00:18 ago on Thu 30 Mar 2023 02:22:16 PM CEST.
Dependencies resolved.
========================================================================================================
 Package                    Architecture         Version                     Repository            Size
========================================================================================================
Upgrading:
 shadow-utils               x86_64               2:4.13-6.fc38               fedora               1.3 M

Transaction Summary
========================================================================================================
Upgrade  1 Package

Total download size: 1.3 M
Is this ok [y/N]: y
Downloading Packages:
shadow-utils-4.13-6.fc38.x86_64.rpm                                     9.0 MB/s | 1.3 MB     00:00    
--------------------------------------------------------------------------------------------------------
Total                                                                   2.7 MB/s | 1.3 MB     00:00     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                1/1 
  Upgrading        : shadow-utils-2:4.13-6.fc38.x86_64                                              1/2 
Error unpacking rpm package shadow-utils-2:4.13-6.fc38.x86_64
  Verifying        : shadow-utils-2:4.13-6.fc38.x86_64                                              1/2 
  Verifying        : shadow-utils-2:4.13-4.fc38.x86_64                                              2/2 

Failed:
  shadow-utils-2:4.13-4.fc38.x86_64                  shadow-utils-2:4.13-6.fc38.x86_64                 

Error: Transaction failed

$ sudo dnf check

$ rpm -q shadow-utils
shadow-utils-4.13-4.fc38.x86_64

And, as requested:
$ sudo dnf info shadow-utils
Last metadata expiration check: 0:00:36 ago on Thu 30 Mar 2023 02:22:16 PM CEST.
Installed Packages
Name         : shadow-utils
Epoch        : 2
Version      : 4.13
Release      : 4.fc38
Architecture : x86_64
Size         : 3.9 M
Source       : shadow-utils-4.13-4.fc38.src.rpm
Repository   : @System
From repo    : anaconda
Summary      : Utilities for managing accounts and shadow password files
URL          : https://github.com/shadow-maint/shadow
License      : BSD-3-Clause AND GPL-2.0-or-later
Description  : The shadow-utils package includes the necessary programs for
             : converting UNIX password files to the shadow password format, plus
             : programs for managing user and group accounts. The pwconv command
             : converts passwords to the shadow password format. The pwunconv command
             : unconverts shadow passwords and generates a passwd file (a standard
             : UNIX password file). The pwck command checks the integrity of password
             : and shadow files. The lastlog command prints out the last login times
             : for all users. The useradd, userdel, and usermod commands are used for
             : managing user accounts. The groupadd, groupdel, and groupmod commands
             : are used for managing group accounts.

Available Packages
Name         : shadow-utils
Epoch        : 2
Version      : 4.13
Release      : 6.fc38
Architecture : x86_64
Size         : 1.3 M
Source       : shadow-utils-4.13-6.fc38.src.rpm
Repository   : fedora
Summary      : Utilities for managing accounts and shadow password files
URL          : https://github.com/shadow-maint/shadow
License      : BSD-3-Clause AND GPL-2.0-or-later
Description  : The shadow-utils package includes the necessary programs for
             : converting UNIX password files to the shadow password format, plus
             : programs for managing user and group accounts. The pwconv command
             : converts passwords to the shadow password format. The pwunconv command
             : unconverts shadow passwords and generates a passwd file (a standard
             : UNIX password file). The pwck command checks the integrity of password
             : and shadow files. The lastlog command prints out the last login times
             : for all users. The useradd, userdel, and usermod commands are used for
             : managing user accounts. The groupadd, groupdel, and groupmod commands
             : are used for managing group accounts.

Name         : shadow-utils
Epoch        : 2
Version      : 4.13
Release      : 6.fc38
Architecture : x86_64
Size         : 1.3 M
Source       : shadow-utils-4.13-6.fc38.src.rpm
Repository   : updates-testing
Summary      : Utilities for managing accounts and shadow password files
URL          : https://github.com/shadow-maint/shadow
License      : BSD-3-Clause AND GPL-2.0-or-later
Description  : The shadow-utils package includes the necessary programs for
             : converting UNIX password files to the shadow password format, plus
             : programs for managing user and group accounts. The pwconv command
             : converts passwords to the shadow password format. The pwunconv command
             : unconverts shadow passwords and generates a passwd file (a standard
             : UNIX password file). The pwck command checks the integrity of password
             : and shadow files. The lastlog command prints out the last login times
             : for all users. The useradd, userdel, and usermod commands are used for
             : managing user accounts. The groupadd, groupdel, and groupmod commands
             : are used for managing group accounts.

Comment 3 Kamil Páral 2023-03-30 12:33:38 UTC
Inside the toolbox:

$ sudo dnf download shadow-utils
shadow-utils-4.13-6.fc38.x86_64.rpm                                     9.6 MB/s | 1.3 MB     00:00    

$ sudo rpm -Uv ./shadow-utils-4.13-6.fc38.x86_64.rpm 
Verifying packages...
Preparing packages...
shadow-utils-2:4.13-6.fc38.x86_64
error: unpacking of archive failed on file /usr/bin/newgidmap;64258170: cpio: cap_set_file failed - No data available
error: shadow-utils-2:4.13-6.fc38.x86_64: install failed
error: shadow-utils-2:4.13-4.fc38.x86_64: erase skipped

Comment 4 Iker Pedrosa 2023-03-30 13:57:48 UTC
I don't know if there's any relationship but can you share the toolbox version?
dnf info toolbox
Last metadata expiration check: 0:00:11 ago on jue 30 mar 2023 15:56:47.
Installed Packages
Name         : toolbox
Version      : 0.0.99.4
Release      : 1.fc37
Architecture : x86_64
Size         : 7.4 M
Source       : toolbox-0.0.99.4-1.fc37.src.rpm
Repository   : @System
From repo    : updates
Summary      : Tool for containerized command line environments on Linux
URL          : https://containertoolbx.org/
License      : ASL 2.0
Description  : Toolbox is a tool for Linux operating systems, which allows the use of
             : containerized command line environments. It is built on top of Podman and
             : other standard container technologies from OCI.

Comment 5 Debarshi Ray 2023-03-30 16:10:37 UTC
> error: unpacking of archive failed on file /usr/bin/newgidmap;
>   64254ae7: cpio: cap_set_file failed - No data available
> error: shadow-utils-2:4.13-6.fc38.x86_64: install failed

/usr/bin/newuidmap and /usr/bin/newgidmap have some file capabilities set:
  $ getcap /usr/bin/newuidmap 
  /usr/bin/newuidmap cap_setuid=ep
  $ getcap /usr/bin/newgidmap 
  /usr/bin/newgidmap cap_setgid=ep

I wonder if that's causing some problems here.

Can you reproduce this in a Fedora 37 or 36 container?

Comment 6 Debarshi Ray 2023-03-30 16:16:08 UTC
I was able to update shadow-utils inside a Fedora 36 container:
  ⬢$ sudo rpm -Uvh https://kojipkgs.fedoraproject.org//packages/shadow-utils/4.11.1/6.fc36/x86_64/shadow-utils-4.11.1-6.fc36.x86_64.rpm
  Retrieving https://kojipkgs.fedoraproject.org//packages/shadow-utils/4.11.1/6.fc36/x86_64/shadow-utils-4.11.1-6.fc36.x86_64.rpm
  Verifying...                          ################################# [100%]
  Preparing...                          ################################# [100%]
  Updating / installing...
     1:shadow-utils-2:4.11.1-6.fc36     ################################# [ 50%]
  Cleaning up / removing...
     2:shadow-utils-2:4.11.1-5.fc36     ################################# [100%]

... that was created with Podman 4.4.1:
  ⬢$ cat /run/.containerenv 
  engine="podman-4.4.1"
  name="fedora-toolbox-36"
  id="159cec18365e1d1335f5c69337038997279016c898ce83c2a51e5fb0588f67e4"
  image="registry.fedoraproject.org/fedora-toolbox:36"
  imageid="4ba70524970ebd4187eccb557f95a7737c6dde3e408f3dc4ea4dbd9a40f7a1ff"
  rootless=1
  graphRootMounted=1

Comment 7 Debarshi Ray 2023-03-30 16:17:28 UTC
I remember that Podman 4.0.3 got stricter about capabilities inside containers because of CVE-2022-27649 but I clearly have a newer Podman.

Comment 8 Boyd 2023-03-30 17:08:45 UTC
I thought it also may be a stale local fedora-toolbox-38 image but still have the issue after the following:


````
[bkelly@xps ~]$ toolbox --version
toolbox version 0.0.99.4


toolbox list;
901c633cace2  registry.fedoraproject.org/fedora-toolbox:38  4 weeks ago

podman rmi  901c633cace2
````
then:
````
podman create   (defaults to 38 and downloads new image)
toolbox enter   (defaults to 38)

su -
dnf update -y

  vim-minimal-2:9.0.1429-1.fc38.x86_64                       vte-profile-0.72.0-1.fc38.x86_64                           
  which-2.21-39.fc38.x86_64                                 
Failed:
  shadow-utils-2:4.13-4.fc38.x86_64                          shadow-utils-2:4.13-6.fc38.x86_64                         

````

Comment 9 Boyd 2023-03-30 17:12:50 UTC
Sorry I meant above instead of podman create: toolbox create (defaults to 38 and downloads new image)

Comment 10 Iker Pedrosa 2023-03-31 06:48:33 UTC
Well, I run out of ideas here. If the toolbox version (0.0.99.4), the image (901c633cace2) and the shadow-utils version (shadow-utils-2:4.13-6.fc38.x86_64) are the same, then I think this should behave in the same way.

I don't think this is a shadow-utils problem because the capabilities setting that is failing has been there for years, but I have no clue as to where the problem lies. Does anybody have any idea?

Comment 11 Kamil Páral 2023-03-31 08:49:00 UTC
This only happens on F38 host. It's not related to the container image running (I can reproduce it on F36, F37 and F38 containers on an F38 host). It's also probably not directly related to shadow-utils, that just triggers the problem, and likely there are other packages that will behave the same.

On my F38, I have:
podman-4.4.3-1.fc38.x86_64
toolbox-0.0.99.4-1.fc38.x86_64
kernel-6.2.8-300.fc38.x86_64

Please note that the F38 container image got refreshed the last night, so the shadow-utils update is no longer available, it's included in the image. But reinstalling the package triggers the bug as well.

Reproducer:
1. Use an F38 host
2. toolbox create --release 38 f38toolbox  # or you can use 37 or 36, doesn't matter
3. toolbox enter f38toolbox
4. sudo dnf reinstall shadow-utils  # or `dnf update`, depending whether there is currently a shadow-utils update available or not
5. "Error unpacking rpm package shadow-utils-..."


When I try the same approach on F37 host, it works just fine (I only tried F38 container, but I think that's enough for verification).

F37 has:
podman-4.4.2-2.fc37.x86_64
toolbox-0.0.99.4-1.fc37.x86_64
kernel-6.2.8-200.fc37.x86_64


So either this is in podman-4.4.2 -> podman-4.4.3 change, or something else is specific for F38 (like a different config file or something), or some other package is involved in this. Reassigning to podman.

Comment 12 Kamil Páral 2023-03-31 08:58:26 UTC
Proposing for a blocker discussion.

Comment 13 Boyd 2023-03-31 09:21:41 UTC
I'm assuming this is also related from the registry.fedoraproject.org/fedora:38 image

iputils and mtr packages fail to install.

NOTE:  I made sure to delete the local fedora:38 image before running the command below.


[bkelly@xps]$ podman run --rm -it fedora:38 dnf install iputils mtr -y
Fedora 38 - x86_64                                                                   849 kB/s |  66 MB     01:19    
Fedora 38 openh264 (From Cisco) - x86_64                                             665  B/s | 2.5 kB     00:03    
Fedora Modular 38 - x86_64                                                           292 kB/s | 2.3 MB     00:07    
Fedora 38 - x86_64 - Updates                                                          52  B/s | 257  B     00:04    
Fedora Modular 38 - x86_64 - Updates                                                  83  B/s | 257  B     00:03    
Fedora 38 - x86_64 - Test Updates                                                    617 kB/s | 6.2 MB     00:10    
Fedora Modular 38 - x86_64 - Test Updates                                             88  B/s | 257  B     00:02    
Dependencies resolved.
=====================================================================================================================
 Package                   Architecture             Version                           Repository                Size
=====================================================================================================================
Installing:
 iputils                   x86_64                   20221126-2.fc38                   fedora                   185 k
 mtr                       x86_64                   2:0.95-4.fc38                     fedora                    89 k
Installing dependencies:
 jansson                   x86_64                   2.13.1-6.fc38                     fedora                    44 k

Transaction Summary
=====================================================================================================================
Install  3 Packages

Total download size: 318 k
Installed size: 880 k
Downloading Packages:
(1/3): jansson-2.13.1-6.fc38.x86_64.rpm                                               36 kB/s |  44 kB     00:01    
(2/3): mtr-0.95-4.fc38.x86_64.rpm                                                     73 kB/s |  89 kB     00:01    
(3/3): iputils-20221126-2.fc38.x86_64.rpm                                            121 kB/s | 185 kB     00:01    
---------------------------------------------------------------------------------------------------------------------
Total                                                                                102 kB/s | 318 kB     00:03     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                             1/1 
  Installing       : jansson-2.13.1-6.fc38.x86_64                                                                1/3 
  Installing       : mtr-2:0.95-4.fc38.x86_64                                                                    2/3 
Error unpacking rpm package mtr-2:0.95-4.fc38.x86_64
  Installing       : iputils-20221126-2.fc38.x86_64                                                              3/3 
error: unpacking of archive failed on file /usr/sbin/mtr-packet;6426a36b: cpio: cap_set_file failed - Directory not empty
error: mtr-2:0.95-4.fc38.x86_64: install failed

Error unpacking rpm package iputils-20221126-2.fc38.x86_64
  Running scriptlet: iputils-20221126-2.fc38.x86_64                                                              3/3 
error: unpacking of archive failed on file /usr/bin/arping;6426a36b: cpio: cap_set_file failed - No data available
error: iputils-20221126-2.fc38.x86_64: install failed

  Verifying        : iputils-20221126-2.fc38.x86_64                                                              1/3 
  Verifying        : jansson-2.13.1-6.fc38.x86_64                                                                2/3 
  Verifying        : mtr-2:0.95-4.fc38.x86_64                                                                    3/3 

Installed:
  jansson-2.13.1-6.fc38.x86_64                                                                                       
Failed:
  iputils-20221126-2.fc38.x86_64                               mtr-2:0.95-4.fc38.x86_64                              

Error: Transaction failed

Comment 14 Debarshi Ray 2023-03-31 10:47:16 UTC
Have you already tried to reproduce this with just podman(1) without going through toolbox(1) ?

You can find out how a Toolbx container was created and entered by using the 'toolbox --verbose' option with the 'create' and 'enter' commands.  If the container already exists, you can use 'podman inspect' to find out how it was created:
  $ podman inspect --type container <NAME> | less

... and look for 'CreateCommand' in the JSON.

For a rough and ready test, you could try to use 'podman run', instead of the more verbose 'podman create', 'podman start' and 'podman exec' that toolbox(1) uses.  For example:
  $ podman run --env TERM=$TERM --interactive --privileged --rm --security-opt label=disable --tty --userns keep-id registry.fedoraproject.org/fedora:38 /bin/bash

I just chose a random subset of options that toolbox(1) uses.  Feel free to add more flags used by toolbox(1) or remove some, depending on what you find.

Comment 15 Kamil Páral 2023-03-31 10:54:01 UTC
Thanks for a guide, Debarshi. My understanding is that Boyd reproduced in directly in podman in comment 13. Is that satisfactory for this case, or should I try myself?

Comment 16 Debarshi Ray 2023-03-31 10:56:20 UTC
Oh, sorry, I overlooked it.  Yes, that's enough then.

Comment 17 Daniel Walsh 2023-03-31 17:34:20 UTC
Could this be an issue with BTRFS? Does this work for rootful containers?

Could you give the output of podman info?

Comment 18 Giuseppe Scrivano 2023-03-31 18:18:37 UTC
it works for me with XFS at least.

Could you strace the process to see what is exactly failing?

Once you enter the toolbox, go from another terminal and find the process that is running inside the toolbox.  When you have its PID you can do:

$ strace -Z -o /tmp/log -s 10000  -f -p $PID_YOU_HAVE_FOUND_BEFORE

Once strace is running, go to the toolbox session and try to reproduce the issue.  Once that happens, you can terminate strace.  Please attach here the /tmp/log file you've got.

Comment 19 Daniel Walsh 2023-04-01 11:09:58 UTC
I agree this works for me also.

podman info | grep xfs
    Backing Filesystem: xfs


This is most likely a btrfs issue. perhaps interacting with overlayfs,  Which is why I want someone to try as rootful.

Comment 20 Daniel Walsh 2023-04-01 11:18:52 UTC
To prove this is not a Podman or Buildah issue, could you try:

$ buildah unshare
# ctr=$(buildah from scratch)
# mnt=$(buildah mount $ctr)
# dnf install iputils mtr -y --installroot=$mnt --releasever=38
...


And see if the same thing happens, if/when it does than it proves that this is an issue with a combination of user namespace, overlayfs and the underlying file system (I suspect BTRFS).

When running these commands buildah and podman are not in the picture.

Comment 21 Kamil Páral 2023-04-03 12:08:22 UTC
Alright, this is even worse than expected. If you try to update/reinstall shadow-utils, you can't enter the container again, if it was stopped in the meantime.

1. toolbox enter f38toolbox
2. dnf reinstall shadow-utils  # transaction failed
3. logout
4. podman stop f38toolbox
5. toolbox enter f38toolbox

Fails with:
dub 03 13:40:02 hydra f38toolbox[20460]: passwd: libuser initialization error: could not open configuration file `/etc/login.defs': No such file or directory
dub 03 13:40:02 hydra f38toolbox[20460]: Error: failed to remove password for user kparal: failed to invoke passwd(1)


> Could this be an issue with BTRFS?

Huh. I do have btrfs. But I just tested the same steps in a virtual machine with F38 Workstation using btrfs, and it works just fine in there. So my previous statement about this bug not being present on F37 might not be true, because I tested it in a virtual machine. Instead, my VMs seem to be not affected. The filesystem seems not to be the deciding factor. I'll continue investigating.

Comment 22 Kamil Páral 2023-04-03 12:09:07 UTC
Created attachment 1955443 [details]
podman info

(In reply to Daniel Walsh from comment #17)
> Could you give the output of podman info?

Attached.

Comment 23 Kamil Páral 2023-04-03 12:48:23 UTC
I've just install F38 Workstation (with btrfs) on a different bare metal machine, and this toolbox problem doesn't occur. So root cause is not that simple, perhaps it might be related to upgraded systems. I'm trying to test that.

(In reply to Giuseppe Scrivano from comment #18)
> Could you strace the process to see what is exactly failing?
> 
> Once you enter the toolbox, go from another terminal and find the process
> that is running inside the toolbox.

Sorry, which process? The "dnf reinstall" process once started?

Comment 24 Kamil Páral 2023-04-03 13:10:34 UTC
I upgraded F37->F38 Workstation in VM and this problem is still not present, so it's not as easy as doing a system upgrade.

(In reply to Daniel Walsh from comment #20)
> $ buildah unshare
> # ctr=$(buildah from scratch)
> # mnt=$(buildah mount $ctr)
> # dnf install iputils mtr -y --installroot=$mnt --releasever=38

The same error happens. So, how do we debug this next and where do we assign it?

(Also, how do I delete this temporary container, when it doesn't show up in `podman ps -a`? Thanks).

Comment 25 Debarshi Ray 2023-04-03 13:45:39 UTC
(In reply to Kamil Páral from comment #24)
> (Also, how do I delete this temporary container, when it doesn't show up in
> `podman ps -a`? Thanks).

'buildah from ...' creates a so-called 'working container', which is different from 'container', when you think in terms of buildah(1) versus podman(1).  A 'working container' is a temporary container that's used to build an image (eg., from a Container/Dockerfile).  If you look really closely the manuals do use the right terminology, but it's hard to spot the difference at first.

So, instead of 'podman ps ...', try 'buildah containers' and 'buildah rm'.

Comment 26 Nalin Dahyabhai 2023-04-03 15:34:51 UTC
The `podman info` in https://bugzilla.redhat.com/attachment.cgi?id=1955443 includes this nugget:
```
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "false"
```
That, combined with the kernel version (6.2.9-300.fc38.x86_64), where we know the kernel's overlay would support native diff, strongly suggests that fuse-overlayfs is involved, and indeed I can trigger (or not) the problem by manipulating the contents of `.local/share/containers/storage/overlay/.has-mount-program` before running the reproducer command.

A faster way to trigger errors setting file capabilities might be `rpm --setcaps shadow-utils`, which doesn't need to hit the network.
Running rpm in a container under strace with native overlay, rpm calling fsetxattr("security.capability") is succeeding.
Running rpm in a container under strace with fuse-overlay, rpm calling fsetxattr("security.capability") returns EOPNOTSUPP, and I can see fuse-overlayfs making the same call and getting the same result back from the kernel at the same time.  It's not obvious to me why fuse-overlayfs is getting that error, though, because its process appears to be running with CAP_SYS_ADMIN.

Comment 27 Kamil Páral 2023-04-03 15:45:41 UTC
Thanks, Nalin. I can confirm my system has "Native Overlay Diff: false" and ~/.local/share/containers/storage/overlay/.has-mount-program is "true", while my VMs, where everything works, have completely opposite values. If I write "false" to .has-mount-program, the VM suddenly exhibits the same problem as I have on my main system.

Comment 28 Geoffrey Marr 2023-04-03 20:26:31 UTC
Discussed during the 2023-04-03 blocker review meeting: [0]

The decision to classify this bug as a "RejectedBlocker (Final)" and an "AcceptedFreezeException (Final)" was made as we really do not have the framework in place to treat toolbox as release blocking for now (see tickets). It's accepted as an FE though, as it would definitely be good to have it working at release for the increasingly popular immutable editions.

[0] https://meetbot.fedoraproject.org/fedora-blocker-review/2023-04-03/f38-blocker-review.2023-04-03-16.01.txt

Comment 29 Kamil Páral 2023-04-05 06:58:53 UTC
Proposing as a Prioritized bug. This affects just some users (probably those who used podman/toolbox with some older kernel) and not clean installs, but their containers will go kaboom once there's a certain system update. This is very bad.

Comment 30 Giuseppe Scrivano 2023-04-05 10:13:10 UTC
it seems a regression in the kernel or libfuse.

With the following packages:

$ rpm -q fuse3
fuse3-3.13.1-1.fc38.x86_64
$ uname -r
6.2.9-300.fc38.x86_64

I see:

$ podman unshare strace -f -Z podman run --rm  fedora rpm --setcaps shadow-utils 2>&1 | grep setxattr
[pid 16209] fsetxattr(33, ".capability", "\1\0\0\3@\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0", 24, 0) = -1 EOPNOTSUPP (Operation not supported)
[pid 16215] fsetxattr(9, "security.capability", "\1\0\0\2@\0\0\0\0\0\0\0\0\0\0\0\0\0\0", 20, 0) = -1 EOPNOTSUPP (Operation not supported)

the first fsetxattr is what fuse-overlayfs does (and I've verified it just pass the original value it gets).


fuse-overlayfs gets a request to setxattr to ".capability" (note the missing security part).  While on F37 with the following packages:

$ rpm -q fuse3
fuse3-3.10.5-5.fc37.x86_64

$ uname -r
6.2.7-200.fc37.x86_64

it works fine.



For a simpler reproducer, I've adapted the fuse passthrough example:

/////
/*
  FUSE: Filesystem in Userspace
  Copyright (C) 2001-2007  Miklos Szeredi <miklos>
  Copyright (C) 2011       Sebastian Pipping <sebastian>
  This program can be distributed under the terms of the GNU GPLv2.
  See the file COPYING.
*/

/** @file
 *
 * This file system mirrors the existing file system hierarchy of the
 * system, starting at the root file system. This is implemented by
 * just "passing through" all requests to the corresponding user-space
 * libc functions. Its performance is terrible.
 *
 * Compile with
 *
 *     gcc -Wall passthrough.c `pkg-config fuse3 --cflags --libs` -o passthrough
 *
 * ## Source code ##
 * \include passthrough.c
 */


#define FUSE_USE_VERSION 31

#define _GNU_SOURCE

#ifdef linux
/* For pread()/pwrite()/utimensat() */
#define _XOPEN_SOURCE 700
#endif

#define HAVE_SETXATTR 1

#include <fuse.h>
#include <stdio.h>
#include <string.h>
#include <unistd.h>
#include <fcntl.h>
#include <sys/stat.h>
#include <dirent.h>
#include <errno.h>
#ifdef __FreeBSD__
#include <sys/socket.h>
#include <sys/un.h>
#endif
#include <sys/time.h>
#ifdef HAVE_SETXATTR
#include <sys/xattr.h>
#endif

static int mknod_wrapper(int dirfd, const char *path, const char *link,
	int mode, dev_t rdev)
{
	int res;

	if (S_ISREG(mode)) {
		res = openat(dirfd, path, O_CREAT | O_EXCL | O_WRONLY, mode);
		if (res >= 0)
			res = close(res);
	} else if (S_ISDIR(mode)) {
		res = mkdirat(dirfd, path, mode);
	} else if (S_ISLNK(mode) && link != NULL) {
		res = symlinkat(link, dirfd, path);
	} else if (S_ISFIFO(mode)) {
		res = mkfifoat(dirfd, path, mode);
#ifdef __FreeBSD__
	} else if (S_ISSOCK(mode)) {
		struct sockaddr_un su;
		int fd;

		if (strlen(path) >= sizeof(su.sun_path)) {
			errno = ENAMETOOLONG;
			return -1;
		}
		fd = socket(AF_UNIX, SOCK_STREAM, 0);
		if (fd >= 0) {
			/*
			 * We must bind the socket to the underlying file
			 * system to create the socket file, even though
			 * we'll never listen on this socket.
			 */
			su.sun_family = AF_UNIX;
			strncpy(su.sun_path, path, sizeof(su.sun_path));
			res = bindat(dirfd, fd, (struct sockaddr*)&su,
				sizeof(su));
			if (res == 0)
				close(fd);
		} else {
			res = -1;
		}
#endif
	} else {
		res = mknodat(dirfd, path, mode, rdev);
	}

	return res;
}

static int fill_dir_plus = 0;

static void *xmp_init(struct fuse_conn_info *conn,
		      struct fuse_config *cfg)
{
	(void) conn;
	cfg->use_ino = 1;

	/* Pick up changes from lower filesystem right away. This is
	   also necessary for better hardlink support. When the kernel
	   calls the unlink() handler, it does not know the inode of
	   the to-be-removed entry and can therefore not invalidate
	   the cache of the associated inode - resulting in an
	   incorrect st_nlink value being reported for any remaining
	   hardlinks to this inode. */
	cfg->entry_timeout = 0;
	cfg->attr_timeout = 0;
	cfg->negative_timeout = 0;

	return NULL;
}

static int xmp_getattr(const char *path, struct stat *stbuf,
		       struct fuse_file_info *fi)
{
	(void) fi;
	int res;

	res = lstat(path, stbuf);
	if (res == -1)
		return -errno;

	return 0;
}

static int xmp_access(const char *path, int mask)
{
	int res;

	res = access(path, mask);
	if (res == -1)
		return -errno;

	return 0;
}

static int xmp_readlink(const char *path, char *buf, size_t size)
{
	int res;

	res = readlink(path, buf, size - 1);
	if (res == -1)
		return -errno;

	buf[res] = '\0';
	return 0;
}


static int xmp_readdir(const char *path, void *buf, fuse_fill_dir_t filler,
		       off_t offset, struct fuse_file_info *fi,
		       enum fuse_readdir_flags flags)
{
	DIR *dp;
	struct dirent *de;

	(void) offset;
	(void) fi;
	(void) flags;

	dp = opendir(path);
	if (dp == NULL)
		return -errno;

	while ((de = readdir(dp)) != NULL) {
		struct stat st;
		memset(&st, 0, sizeof(st));
		st.st_ino = de->d_ino;
		st.st_mode = de->d_type << 12;
		if (filler(buf, de->d_name, &st, 0, fill_dir_plus))
			break;
	}

	closedir(dp);
	return 0;
}

static int xmp_mknod(const char *path, mode_t mode, dev_t rdev)
{
	int res;

	res = mknod_wrapper(AT_FDCWD, path, NULL, mode, rdev);
	if (res == -1)
		return -errno;

	return 0;
}

static int xmp_mkdir(const char *path, mode_t mode)
{
	int res;

	res = mkdir(path, mode);
	if (res == -1)
		return -errno;

	return 0;
}

static int xmp_unlink(const char *path)
{
	int res;

	res = unlink(path);
	if (res == -1)
		return -errno;

	return 0;
}

static int xmp_rmdir(const char *path)
{
	int res;

	res = rmdir(path);
	if (res == -1)
		return -errno;

	return 0;
}

static int xmp_symlink(const char *from, const char *to)
{
	int res;

	res = symlink(from, to);
	if (res == -1)
		return -errno;

	return 0;
}

static int xmp_rename(const char *from, const char *to, unsigned int flags)
{
	int res;

	if (flags)
		return -EINVAL;

	res = rename(from, to);
	if (res == -1)
		return -errno;

	return 0;
}

static int xmp_link(const char *from, const char *to)
{
	int res;

	res = link(from, to);
	if (res == -1)
		return -errno;

	return 0;
}

static int xmp_chmod(const char *path, mode_t mode,
		     struct fuse_file_info *fi)
{
	(void) fi;
	int res;

	res = chmod(path, mode);
	if (res == -1)
		return -errno;

	return 0;
}

static int xmp_chown(const char *path, uid_t uid, gid_t gid,
		     struct fuse_file_info *fi)
{
	(void) fi;
	int res;

	res = lchown(path, uid, gid);
	if (res == -1)
		return -errno;

	return 0;
}

static int xmp_truncate(const char *path, off_t size,
			struct fuse_file_info *fi)
{
	int res;

	if (fi != NULL)
		res = ftruncate(fi->fh, size);
	else
		res = truncate(path, size);
	if (res == -1)
		return -errno;

	return 0;
}

#ifdef HAVE_UTIMENSAT
static int xmp_utimens(const char *path, const struct timespec ts[2],
		       struct fuse_file_info *fi)
{
	(void) fi;
	int res;

	/* don't use utime/utimes since they follow symlinks */
	res = utimensat(0, path, ts, AT_SYMLINK_NOFOLLOW);
	if (res == -1)
		return -errno;

	return 0;
}
#endif

static int xmp_create(const char *path, mode_t mode,
		      struct fuse_file_info *fi)
{
	int res;

	res = open(path, fi->flags, mode);
	if (res == -1)
		return -errno;

	fi->fh = res;
	return 0;
}

static int xmp_open(const char *path, struct fuse_file_info *fi)
{
	int res;

	res = open(path, fi->flags);
	if (res == -1)
		return -errno;

	fi->fh = res;
	return 0;
}

static int xmp_read(const char *path, char *buf, size_t size, off_t offset,
		    struct fuse_file_info *fi)
{
	int fd;
	int res;

	if(fi == NULL)
		fd = open(path, O_RDONLY);
	else
		fd = fi->fh;
	
	if (fd == -1)
		return -errno;

	res = pread(fd, buf, size, offset);
	if (res == -1)
		res = -errno;

	if(fi == NULL)
		close(fd);
	return res;
}

static int xmp_write(const char *path, const char *buf, size_t size,
		     off_t offset, struct fuse_file_info *fi)
{
	int fd;
	int res;

	(void) fi;
	if(fi == NULL)
		fd = open(path, O_WRONLY);
	else
		fd = fi->fh;
	
	if (fd == -1)
		return -errno;

	res = pwrite(fd, buf, size, offset);
	if (res == -1)
		res = -errno;

	if(fi == NULL)
		close(fd);
	return res;
}

static int xmp_statfs(const char *path, struct statvfs *stbuf)
{
	int res;

	res = statvfs(path, stbuf);
	if (res == -1)
		return -errno;

	return 0;
}

static int xmp_release(const char *path, struct fuse_file_info *fi)
{
	(void) path;
	close(fi->fh);
	return 0;
}

static int xmp_fsync(const char *path, int isdatasync,
		     struct fuse_file_info *fi)
{
	/* Just a stub.	 This method is optional and can safely be left
	   unimplemented */

	(void) path;
	(void) isdatasync;
	(void) fi;
	return 0;
}

#ifdef HAVE_POSIX_FALLOCATE
static int xmp_fallocate(const char *path, int mode,
			off_t offset, off_t length, struct fuse_file_info *fi)
{
	int fd;
	int res;

	(void) fi;

	if (mode)
		return -EOPNOTSUPP;

	if(fi == NULL)
		fd = open(path, O_WRONLY);
	else
		fd = fi->fh;
	
	if (fd == -1)
		return -errno;

	res = -posix_fallocate(fd, offset, length);

	if(fi == NULL)
		close(fd);
	return res;
}
#endif

#ifdef HAVE_SETXATTR
/* xattr operations are optional and can safely be left unimplemented */
static int xmp_setxattr(const char *path, const char *name, const char *value,
			size_t size, int flags)
{
	int res;

	printf("setxattr %s\n", name);
	res = lsetxattr(path, name, value, size, flags);
	if (res == -1)
		return -errno;
	return 0;
}

static int xmp_getxattr(const char *path, const char *name, char *value,
			size_t size)
{
	int res = lgetxattr(path, name, value, size);
	if (res == -1)
		return -errno;
	return res;
}

static int xmp_listxattr(const char *path, char *list, size_t size)
{
	int res = llistxattr(path, list, size);
	if (res == -1)
		return -errno;
	return res;
}

static int xmp_removexattr(const char *path, const char *name)
{
	int res = lremovexattr(path, name);
	if (res == -1)
		return -errno;
	return 0;
}
#endif /* HAVE_SETXATTR */

#ifdef HAVE_COPY_FILE_RANGE
static ssize_t xmp_copy_file_range(const char *path_in,
				   struct fuse_file_info *fi_in,
				   off_t offset_in, const char *path_out,
				   struct fuse_file_info *fi_out,
				   off_t offset_out, size_t len, int flags)
{
	int fd_in, fd_out;
	ssize_t res;

	if(fi_in == NULL)
		fd_in = open(path_in, O_RDONLY);
	else
		fd_in = fi_in->fh;

	if (fd_in == -1)
		return -errno;

	if(fi_out == NULL)
		fd_out = open(path_out, O_WRONLY);
	else
		fd_out = fi_out->fh;

	if (fd_out == -1) {
		close(fd_in);
		return -errno;
	}

	res = copy_file_range(fd_in, &offset_in, fd_out, &offset_out, len,
			      flags);
	if (res == -1)
		res = -errno;

	if (fi_out == NULL)
		close(fd_out);
	if (fi_in == NULL)
		close(fd_in);

	return res;
}
#endif

static off_t xmp_lseek(const char *path, off_t off, int whence, struct fuse_file_info *fi)
{
	int fd;
	off_t res;

	if (fi == NULL)
		fd = open(path, O_RDONLY);
	else
		fd = fi->fh;

	if (fd == -1)
		return -errno;

	res = lseek(fd, off, whence);
	if (res == -1)
		res = -errno;

	if (fi == NULL)
		close(fd);
	return res;
}

static const struct fuse_operations xmp_oper = {
	.init           = xmp_init,
	.getattr	= xmp_getattr,
	.access		= xmp_access,
	.readlink	= xmp_readlink,
	.readdir	= xmp_readdir,
	.mknod		= xmp_mknod,
	.mkdir		= xmp_mkdir,
	.symlink	= xmp_symlink,
	.unlink		= xmp_unlink,
	.rmdir		= xmp_rmdir,
	.rename		= xmp_rename,
	.link		= xmp_link,
	.chmod		= xmp_chmod,
	.chown		= xmp_chown,
	.truncate	= xmp_truncate,
#ifdef HAVE_UTIMENSAT
	.utimens	= xmp_utimens,
#endif
	.open		= xmp_open,
	.create 	= xmp_create,
	.read		= xmp_read,
	.write		= xmp_write,
	.statfs		= xmp_statfs,
	.release	= xmp_release,
	.fsync		= xmp_fsync,
#ifdef HAVE_POSIX_FALLOCATE
	.fallocate	= xmp_fallocate,
#endif
#ifdef HAVE_SETXATTR
	.setxattr	= xmp_setxattr,
	.getxattr	= xmp_getxattr,
	.listxattr	= xmp_listxattr,
	.removexattr	= xmp_removexattr,
#endif
#ifdef HAVE_COPY_FILE_RANGE
	.copy_file_range = xmp_copy_file_range,
#endif
	.lseek		= xmp_lseek,
};

int main(int argc, char *argv[])
{
	enum { MAX_ARGS = 10 };
	int i,new_argc;
	char *new_argv[MAX_ARGS];

	umask(0);
			/* Process the "--plus" option apart */
	for (i=0, new_argc=0; (i<argc) && (new_argc<MAX_ARGS); i++) {
		if (!strcmp(argv[i], "--plus")) {
			fill_dir_plus = FUSE_FILL_DIR_PLUS;
		} else {
			new_argv[new_argc++] = argv[i];
		}
	}
	return fuse_main(new_argc, new_argv, &xmp_oper, NULL);
}
////////

$ gcc -o /tmp/fuse-pass fuse-pass.c -I /usr/include/fuse3/ -l fuse3

$ unshare -rm tmux
$ mkdir /tmp/foo
$ /tmp/fuse-pass -f /tmp/foo/
Ignoring invalid max threads value 4294967295 > max (100000).
setxattr urity.capability
setxattr urity.capability

(from another terminal in tmux, so in the same user+mount namespace):
$ touch /tmp/file ; attr -s 'security.capability' -V foo /tmp/foo/tmp/file

That triggers the two lines "setxattr" in the first terminal, which has clearly the wrong value.

For the record, on Fedora 37, the output looks like the following:

setxattr user.security.capability
setxattr user.security.capability

Comment 31 Giuseppe Scrivano 2023-04-06 12:23:17 UTC
changing the component to fuse3 as Miklos confirmed the issue there.  Should be fixed in 3.14.1

Comment 32 Debarshi Ray 2023-04-06 12:31:37 UTC
Thanks for all the detective work, folks!

Comment 33 Fedora Update System 2023-04-11 23:29:43 UTC
FEDORA-2023-37e142ea83 has been submitted as an update to Fedora 38. https://bodhi.fedoraproject.org/updates/FEDORA-2023-37e142ea83

Comment 34 Adam Williamson 2023-04-12 00:32:09 UTC
I backported the commit that I believe addresses this - https://github.com/libfuse/libfuse/commit/024eccbfa5c71a1c927e83753c26c1e6a738708f - in the update. I think the state of the art on this upstream is https://github.com/libfuse/libfuse/commit/681a0c1178fa93017a363a901d0348710582e90b , which is after 3.14.1, but I didn't want to do any more change than necessary on top of 3.13.1 for F38.

If folks could let us know if the update solves the problem, that'd be great.

Comment 35 Fedora Update System 2023-04-12 03:00:25 UTC
FEDORA-2023-37e142ea83 has been pushed to the Fedora 38 testing repository.

You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2023-37e142ea83

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 36 Kamil Páral 2023-04-12 06:55:08 UTC
(In reply to Fedora Update System from comment #33)
> FEDORA-2023-37e142ea83 has been submitted as an update to Fedora 38.
> https://bodhi.fedoraproject.org/updates/FEDORA-2023-37e142ea83

I tested my production machine with 'Native Overlay Diff: "false"' and a VM with 'Native Overlay Diff: "true"'. In both cases, I can now reinstall shadow-utils/mtr/iptables in a F38 container without issues. I believe the issue is fixed. Thanks!

Comment 37 Fedora Update System 2023-04-13 20:09:40 UTC
FEDORA-2023-37e142ea83 has been pushed to the Fedora 38 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 38 Boyd 2023-04-17 15:58:46 UTC
Not 100% sure if this is related as the error message is different, but still not able to install shadow-utils on the fedora-minimal:38 image.  Output as follows:

[bkelly@xps cmp]$ podman rmi fedora-minimal:38
Untagged: registry.fedoraproject.org/fedora-minimal:38
[bkelly@xps cmp]$ podman run -it fedora-minimal:38 microdnf install shadow-utils
Resolved "fedora-minimal" as an alias (/home/bkelly/.cache/containers/short-name-aliases.conf)
Trying to pull registry.fedoraproject.org/fedora-minimal:38...
Getting image source signatures
Copying blob bcacf4632f8d skipped: already exists  
Copying config be1fb23bc1 done  
Writing manifest to image destination
Storing signatures
Updating and loading repositories:
 Fedora 38 - x86_64 - Updates                                                100% | 152.4 KiB/s |   1.7 MiB |  00m11s
 Fedora 38 - x86_64                                                          100% | 143.5 KiB/s |  21.4 MiB |  02m33s
 Fedora 38 openh264 (From Cisco) - x86_64                                    100% |   1.4 KiB/s |   6.5 KiB |  00m05s
Repositories loaded.
Package                                Arch       Version                                Repository              Size
Installing:                                                                                                          
 shadow-utils                          x86_64     2:4.13-6.fc38                          fedora               4.0 MiB
Installing dependencies:                                                                                             
 libeconf                              x86_64     0.4.0-5.fc38                           fedora              45.3 KiB
 libsemanage                           x86_64     3.5-2.fc38                             fedora             297.1 KiB

Transaction Summary:
 Installing:        3 packages

Is this ok [y/N]: y      
Downloading Packages:
[1/3] libeconf-0:0.4.0-5.fc38.x86_64                                         100% |  28.7 KiB/s |  27.0 KiB |  00m01s
[2/3] libsemanage-0:3.5-2.fc38.x86_64                                        100% |  78.1 KiB/s | 119.6 KiB |  00m02s
[3/3] shadow-utils-2:4.13-6.fc38.x86_64                                      100% | 113.3 KiB/s |   1.3 MiB |  00m11s
---------------------------------------------------------------------------------------------------------------------
[3/3] Total                                                                  100% | 108.3 KiB/s |   1.4 MiB |  00m13s

Verifying PGP signatures

Running transaction
[1/2] Verify package files                                                   100% | 187.0   B/s |   3.0   B |  00m00s
[2/3] Prepare transaction                                                    100% | 166.0   B/s |   3.0   B |  00m00s
[3/4] Installing libsemanage-0:3.5-2.fc38.x86_64                             100% |  11.7 MiB/s | 299.0 KiB |  00m00s
[4/5] Installing libeconf-0:0.4.0-5.fc38.x86_64                              100% |   2.2 MiB/s |  46.9 KiB |  00m00s
[5/5] Installing shadow-utils-2:4.13-6.fc38.x86_64                           100% |  59.9 MiB/s |   4.0 MiB |  00m00s
>>> Unpack errro: shadow-utils-2:4.13-6.fc38.x86_64
---------------------------------------------------------------------------------------------------------------------
[5/5] Total                                                                  100% |  17.1 MiB/s |   4.3 MiB |  00m00sTransaction failed: Rpm transaction failed.
[bkelly@xps cmp]$

Comment 39 Kamil Páral 2023-04-18 06:29:41 UTC
(In reply to Boyd from comment #38)
> [bkelly@xps cmp]$ podman run -it fedora-minimal:38 microdnf install

This works on my system. Are you sure you have the update from comment 33 installed? Check the package versions.


Note You need to log in before you can comment on or make changes to this bug.