Bug 2188304 - 'systemd-tmpfiles --create' in Toolbx container gives lots of errors/warnings
Summary: 'systemd-tmpfiles --create' in Toolbx container gives lots of errors/warnings
Keywords:
Status: ASSIGNED
Alias: None
Product: Fedora
Classification: Fedora
Component: toolbox
Version: rawhide
Hardware: x86_64
OS: Linux
unspecified
medium
Target Milestone: ---
Assignee: Debarshi Ray
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2023-04-20 12:12 UTC by Jens Petersen
Modified: 2024-05-07 20:00 UTC (History)
14 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
Type: ---
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github containers toolbox issues 1294 0 None open 'systemd-tmpfiles --create' in Toolbx container gives lots of errors/warnings 2023-05-03 13:25:40 UTC

Description Jens Petersen 2023-04-20 12:12:28 UTC
Installing mock (or systemd packages) inside recent toolbox containers
for Fedora 37 or later seems to give a lot of warnings and errors in
dnf transaction output. AFAICT this started happening in the last few months.

Reproducible: Always

Steps to Reproduce:
1. toolbox create --release 38
2. toolbox enter --release 38
3. sudo dnf install systemd
4. sudo dnf update
Actual Results:  
For 3.:

⬢[petersen@toolbox ~]$ sudo dnf install systemd
Fedora 38 - x86_64                               12 MB/s |  66 MB     00:05    
Fedora 38 openh264 (From Cisco) - x86_64        602  B/s | 2.5 kB     00:04    
Fedora Modular 38 - x86_64                      1.2 MB/s | 2.3 MB     00:01    
Fedora 38 - x86_64 - Updates                    1.7 MB/s | 5.6 MB     00:03    
Fedora Modular 38 - x86_64 - Updates            220  B/s | 257  B     00:01    
Dependencies resolved.
================================================================================
 Package                  Architecture Version               Repository    Size
================================================================================
Installing:
 systemd                  x86_64       253.2-1.fc38          fedora       4.4 M
Installing dependencies:
 device-mapper            x86_64       1.02.189-2.fc38       fedora       139 k
 device-mapper-libs       x86_64       1.02.189-2.fc38       fedora       176 k
 kmod-libs                x86_64       30-4.fc38             fedora        68 k
 libargon2                x86_64       20190702-2.fc38       fedora        28 k
 libseccomp               x86_64       2.5.3-4.fc38          fedora        71 k
 systemd-pam              x86_64       253.2-1.fc38          fedora       345 k
 xkeyboard-config         noarch       2.38-1.fc38           fedora       963 k
Installing weak dependencies:
 cryptsetup-libs          x86_64       2.6.1-1.fc38          fedora       492 k
 libxkbcommon             x86_64       1.5.0-2.fc38          fedora       140 k
 qrencode-libs            x86_64       4.1.1-4.fc38          fedora        61 k
 systemd-networkd         x86_64       253.2-1.fc38          fedora       636 k
 systemd-resolved         x86_64       253.2-1.fc38          fedora       291 k

Transaction Summary
================================================================================
Install  13 Packages

Total download size: 7.8 M
Installed size: 27 M
Is this ok [y/N]: y
Downloading Packages:
(1/13): device-mapper-1.02.189-2.fc38.x86_64.rp 522 kB/s | 139 kB     00:00    
(2/13): device-mapper-libs-1.02.189-2.fc38.x86_ 642 kB/s | 176 kB     00:00    
(3/13): kmod-libs-30-4.fc38.x86_64.rpm          2.7 MB/s |  68 kB     00:00    
(4/13): libargon2-20190702-2.fc38.x86_64.rpm    1.5 MB/s |  28 kB     00:00    
(5/13): cryptsetup-libs-2.6.1-1.fc38.x86_64.rpm 1.6 MB/s | 492 kB     00:00    
(6/13): libseccomp-2.5.3-4.fc38.x86_64.rpm      3.2 MB/s |  71 kB     00:00    
(7/13): libxkbcommon-1.5.0-2.fc38.x86_64.rpm    5.2 MB/s | 140 kB     00:00    
(8/13): qrencode-libs-4.1.1-4.fc38.x86_64.rpm   3.6 MB/s |  61 kB     00:00    
(9/13): systemd-pam-253.2-1.fc38.x86_64.rpm     5.4 MB/s | 345 kB     00:00    
(10/13): systemd-networkd-253.2-1.fc38.x86_64.r 7.4 MB/s | 636 kB     00:00    
(11/13): systemd-resolved-253.2-1.fc38.x86_64.r 5.4 MB/s | 291 kB     00:00    
(12/13): xkeyboard-config-2.38-1.fc38.noarch.rp 8.7 MB/s | 963 kB     00:00    
(13/13): systemd-253.2-1.fc38.x86_64.rpm         12 MB/s | 4.4 MB     00:00    
--------------------------------------------------------------------------------
Total                                           4.1 MB/s | 7.8 MB     00:01     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                        1/1 
  Installing       : xkeyboard-config-2.38-1.fc38.noarch                   1/13 
  Installing       : libxkbcommon-1.5.0-2.fc38.x86_64                      2/13 
  Installing       : qrencode-libs-4.1.1-4.fc38.x86_64                     3/13 
  Installing       : libseccomp-2.5.3-4.fc38.x86_64                        4/13 
  Installing       : libargon2-20190702-2.fc38.x86_64                      5/13 
  Installing       : kmod-libs-30-4.fc38.x86_64                            6/13 
  Installing       : device-mapper-libs-1.02.189-2.fc38.x86_64             7/13 
  Installing       : cryptsetup-libs-2.6.1-1.fc38.x86_64                   8/13 
  Installing       : device-mapper-1.02.189-2.fc38.x86_64                  9/13 
  Installing       : systemd-networkd-253.2-1.fc38.x86_64                 10/13 
  Running scriptlet: systemd-networkd-253.2-1.fc38.x86_64                 10/13 
  Installing       : systemd-pam-253.2-1.fc38.x86_64                      11/13 
  Installing       : systemd-resolved-253.2-1.fc38.x86_64                 12/13 
  Running scriptlet: systemd-resolved-253.2-1.fc38.x86_64                 12/13 
  Installing       : systemd-253.2-1.fc38.x86_64                          13/13 
  Running scriptlet: systemd-253.2-1.fc38.x86_64                          13/13 
Creating group 'input' with GID 104.
Creating group 'kvm' with GID 36.
Creating group 'render' with GID 105.
Creating group 'sgx' with GID 106.
Creating group 'systemd-journal' with GID 190.
Creating group 'systemd-network' with GID 192.
Creating user 'systemd-network' (systemd Network Management) with UID 192 and GID 192.
Creating group 'systemd-oom' with GID 999.
Creating user 'systemd-oom' (systemd Userspace OOM Killer) with UID 999 and GID 999.
Creating group 'systemd-resolve' with GID 193.
Creating user 'systemd-resolve' (systemd Resolver) with UID 193 and GID 193.

  Running scriptlet: systemd-resolved-253.2-1.fc38.x86_64                 13/13 
'/etc/resolv.conf' -> '../run/systemd/resolve/stub-resolv.conf'

  Running scriptlet: systemd-253.2-1.fc38.x86_64                          13/13 
fchownat() of /run/systemd/sessions failed: Operation not permitted
fchownat() of /run/systemd/users failed: Operation not permitted
fchownat() of /var/lib/systemd/coredump failed: Read-only file system
fchownat() of /tmp failed: Operation not permitted
Setting access ACL "u::rwx,g::r-x,g:adm:r-x,g:wheel:r-x,g:4294967295:r-x,g:4294967295:r-x,m::r-x,o::r-x" on /var/log/journal failed: Read-only file system
Failed to re-open '/var/log/journal': Operation not permitted
fchownat() of /var/log/journal failed: Read-only file system
Setting access ACL "u::rwx,g::r-x,g:adm:r-x,g:wheel:r-x,g:4294967295:r-x,g:4294967295:r-x,m::r-x,o::r-x" on /var/log/journal/9392e9ccb14d4745b1e1a97c5f4a9d4c failed: Read-only file system
Failed to re-open '/var/log/journal/9392e9ccb14d4745b1e1a97c5f4a9d4c': Operation not permitted
fchownat() of /var/log/journal/9392e9ccb14d4745b1e1a97c5f4a9d4c failed: Read-only file system
fchownat() of /dev/snd/seq failed: Operation not permitted
fchownat() of /dev/snd/timer failed: Operation not permitted
fchownat() of /dev/loop-control failed: Operation not permitted
fchownat() of /dev/kvm failed: Operation not permitted
fchownat() of /dev/vhost-net failed: Operation not permitted
fchownat() of /dev/vhost-vsock failed: Operation not permitted
Setting access ACL "u::rw-,g::r-x,g:adm:r--,g:wheel:r--,g:4294967295:r-x,g:4294967295:r-x,m::r--,o::---" on /var/log/journal/9392e9ccb14d4745b1e1a97c5f4a9d4c/system.journal failed: Read-only file system
fchownat() of /var/log/journal/9392e9ccb14d4745b1e1a97c5f4a9d4c/system.journal failed: Read-only file system
fchownat() of /sys/kernel/security/ima/binary_runtime_measurements failed: Operation not permitted

/var/tmp/rpm-tmp.Lmq07P: line 4: /usr/lib/systemd/systemd-sysctl: No such file or directory

Failed to reload daemon: Access denied

Failed to start transient service unit: Connection reset by peer
Failed to reload daemon: Transport endpoint is not connected
Failed to start transient service unit: Connection reset by peer
Failed to start jobs: Transport endpoint is not connected

  Verifying        : cryptsetup-libs-2.6.1-1.fc38.x86_64                   1/13 
  Verifying        : device-mapper-1.02.189-2.fc38.x86_64                  2/13 
  Verifying        : device-mapper-libs-1.02.189-2.fc38.x86_64             3/13 
  Verifying        : kmod-libs-30-4.fc38.x86_64                            4/13 
  Verifying        : libargon2-20190702-2.fc38.x86_64                      5/13 
  Verifying        : libseccomp-2.5.3-4.fc38.x86_64                        6/13 
  Verifying        : libxkbcommon-1.5.0-2.fc38.x86_64                      7/13 
  Verifying        : qrencode-libs-4.1.1-4.fc38.x86_64                     8/13 
  Verifying        : systemd-253.2-1.fc38.x86_64                           9/13 
  Verifying        : systemd-networkd-253.2-1.fc38.x86_64                 10/13 
  Verifying        : systemd-pam-253.2-1.fc38.x86_64                      11/13 
  Verifying        : systemd-resolved-253.2-1.fc38.x86_64                 12/13 
  Verifying        : xkeyboard-config-2.38-1.fc38.noarch                  13/13 

Installed:
  cryptsetup-libs-2.6.1-1.fc38.x86_64                                           
  device-mapper-1.02.189-2.fc38.x86_64                                          
  device-mapper-libs-1.02.189-2.fc38.x86_64                                     
  kmod-libs-30-4.fc38.x86_64                                                    
  libargon2-20190702-2.fc38.x86_64                                              
  libseccomp-2.5.3-4.fc38.x86_64                                                
  libxkbcommon-1.5.0-2.fc38.x86_64                                              
  qrencode-libs-4.1.1-4.fc38.x86_64                                             
  systemd-253.2-1.fc38.x86_64                                                   
  systemd-networkd-253.2-1.fc38.x86_64                                          
  systemd-pam-253.2-1.fc38.x86_64                                               
  systemd-resolved-253.2-1.fc38.x86_64                                          
  xkeyboard-config-2.38-1.fc38.noarch                                           

Complete!

4.:

⬢[petersen@toolbox ~]$ sudo dnf update
Last metadata expiration check: 0:14:34 ago on Thu 20 Apr 2023 07:53:04 PM.
Dependencies resolved.
================================================================================
 Package                   Architecture Version             Repository     Size
================================================================================
Upgrading:
 avahi                     x86_64       0.8-22.fc38         updates       300 k
 avahi-libs                x86_64       0.8-22.fc38         updates        66 k
 hwdata                    noarch       0.369-1.fc38        updates       1.6 M
 libwayland-client         x86_64       1.22.0-1.fc38       updates        34 k
 libxml2                   x86_64       2.10.4-1.fc38       updates       701 k
 llvm-libs                 x86_64       16.0.0-2.fc38       fedora         27 M
 man-pages                 noarch       6.04-1.fc38         updates       3.4 M
 mesa-dri-drivers          x86_64       23.0.2-2.fc38       updates        18 M
 mesa-filesystem           x86_64       23.0.2-2.fc38       updates        18 k
 mesa-libglapi             x86_64       23.0.2-2.fc38       updates        54 k
 mesa-va-drivers           x86_64       23.0.2-2.fc38       updates       3.4 M
 mesa-vulkan-drivers       x86_64       23.0.2-2.fc38       updates       9.0 M
 openssh                   x86_64       9.0p1-15.fc38       updates       435 k
 openssh-clients           x86_64       9.0p1-15.fc38       updates       699 k

Transaction Summary
================================================================================
Upgrade  14 Packages

Total download size: 65 M
Is this ok [y/N]: y
Downloading Packages:
(1/14): avahi-libs-0.8-22.fc38.x86_64.rpm       185 kB/s |  66 kB     00:00    
(2/14): avahi-0.8-22.fc38.x86_64.rpm            563 kB/s | 300 kB     00:00    
(3/14): libwayland-client-1.22.0-1.fc38.x86_64. 367 kB/s |  34 kB     00:00    
(4/14): hwdata-0.369-1.fc38.noarch.rpm          2.9 MB/s | 1.6 MB     00:00    
(5/14): libxml2-2.10.4-1.fc38.x86_64.rpm        2.4 MB/s | 701 kB     00:00    
(6/14): llvm-libs-16.0.0-2.fc38.x86_64.rpm       15 MB/s |  27 MB     00:01    
(7/14): man-pages-6.04-1.fc38.noarch.rpm        3.1 MB/s | 3.4 MB     00:01    
(8/14): mesa-filesystem-23.0.2-2.fc38.x86_64.rp  65 kB/s |  18 kB     00:00    
(9/14): mesa-libglapi-23.0.2-2.fc38.x86_64.rpm  568 kB/s |  54 kB     00:00    
(10/14): mesa-va-drivers-23.0.2-2.fc38.x86_64.r 3.1 MB/s | 3.4 MB     00:01    
(11/14): openssh-9.0p1-15.fc38.x86_64.rpm       623 kB/s | 435 kB     00:00    
(12/14): mesa-vulkan-drivers-23.0.2-2.fc38.x86_ 3.7 MB/s | 9.0 MB     00:02    
(13/14): openssh-clients-9.0p1-15.fc38.x86_64.r 975 kB/s | 699 kB     00:00    
(14/14): mesa-dri-drivers-23.0.2-2.fc38.x86_64. 3.9 MB/s |  18 MB     00:04    
--------------------------------------------------------------------------------
Total                                           7.7 MB/s |  65 MB     00:08     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                        1/1 
  Upgrading        : llvm-libs-16.0.0-2.fc38.x86_64                        1/28 
  Upgrading        : mesa-filesystem-23.0.2-2.fc38.x86_64                  2/28 
  Upgrading        : mesa-va-drivers-23.0.2-2.fc38.x86_64                  3/28 
  Upgrading        : openssh-9.0p1-15.fc38.x86_64                          4/28 
  Upgrading        : mesa-libglapi-23.0.2-2.fc38.x86_64                    5/28 
  Upgrading        : libwayland-client-1.22.0-1.fc38.x86_64                6/28 
  Upgrading        : avahi-libs-0.8-22.fc38.x86_64                         7/28 
  Running scriptlet: avahi-0.8-22.fc38.x86_64                              8/28 
  Upgrading        : avahi-0.8-22.fc38.x86_64                              8/28 
  Running scriptlet: avahi-0.8-22.fc38.x86_64                              8/28 
  Upgrading        : mesa-vulkan-drivers-23.0.2-2.fc38.x86_64              9/28 
  Upgrading        : mesa-dri-drivers-23.0.2-2.fc38.x86_64                10/28 
  Upgrading        : openssh-clients-9.0p1-15.fc38.x86_64                 11/28 
  Running scriptlet: openssh-clients-9.0p1-15.fc38.x86_64                 11/28 
  Running scriptlet: man-pages-6.04-1.fc38.noarch                         12/28 
  Upgrading        : man-pages-6.04-1.fc38.noarch                         12/28 
  Running scriptlet: man-pages-6.04-1.fc38.noarch                         12/28 
  Upgrading        : libxml2-2.10.4-1.fc38.x86_64                         13/28 
  Upgrading        : hwdata-0.369-1.fc38.noarch                           14/28 
  Cleanup          : mesa-dri-drivers-23.0.1-1.fc38.x86_64                15/28 
  Cleanup          : mesa-va-drivers-23.0.1-1.fc38.x86_64                 16/28 
  Cleanup          : mesa-vulkan-drivers-23.0.1-1.fc38.x86_64             17/28 
  Running scriptlet: avahi-0.8-20.fc38.x86_64                             18/28 
  Cleanup          : avahi-0.8-20.fc38.x86_64                             18/28 
  Running scriptlet: avahi-0.8-20.fc38.x86_64                             18/28 
Failed to set unit properties on avahi-daemon.service: Access denied
Failed to set unit properties on avahi-daemon.socket: Access denied

  Cleanup          : mesa-filesystem-23.0.1-1.fc38.x86_64                 19/28 
  Running scriptlet: man-pages-6.03-3.fc38.noarch                         20/28 
  Cleanup          : man-pages-6.03-3.fc38.noarch                         20/28 
  Running scriptlet: man-pages-6.03-3.fc38.noarch                         20/28 
  Cleanup          : hwdata-0.368-1.fc38.noarch                           21/28 
  Running scriptlet: openssh-clients-9.0p1-14.fc38.1.x86_64               22/28 
  Cleanup          : openssh-clients-9.0p1-14.fc38.1.x86_64               22/28 
  Cleanup          : openssh-9.0p1-14.fc38.1.x86_64                       23/28 
  Cleanup          : avahi-libs-0.8-20.fc38.x86_64                        24/28 
  Cleanup          : llvm-libs-15.0.7-2.fc38.x86_64                       25/28 
  Cleanup          : libwayland-client-1.21.0-2.fc38.x86_64               26/28 
  Cleanup          : mesa-libglapi-23.0.1-1.fc38.x86_64                   27/28 
  Cleanup          : libxml2-2.10.3-3.fc38.x86_64                         28/28 
  Running scriptlet: libxml2-2.10.3-3.fc38.x86_64                         28/28 
Failed to reload daemon: Access denied

Failed to start transient service unit: Connection reset by peer
Failed to reload daemon: Transport endpoint is not connected
Failed to start transient service unit: Connection reset by peer
Failed to start jobs: Transport endpoint is not connected

  Verifying        : llvm-libs-16.0.0-2.fc38.x86_64                        1/28 
  Verifying        : llvm-libs-15.0.7-2.fc38.x86_64                        2/28 
  Verifying        : avahi-0.8-22.fc38.x86_64                              3/28 
  Verifying        : avahi-0.8-20.fc38.x86_64                              4/28 
  Verifying        : avahi-libs-0.8-22.fc38.x86_64                         5/28 
  Verifying        : avahi-libs-0.8-20.fc38.x86_64                         6/28 
  Verifying        : hwdata-0.369-1.fc38.noarch                            7/28 
  Verifying        : hwdata-0.368-1.fc38.noarch                            8/28 
  Verifying        : libwayland-client-1.22.0-1.fc38.x86_64                9/28 
  Verifying        : libwayland-client-1.21.0-2.fc38.x86_64               10/28 
  Verifying        : libxml2-2.10.4-1.fc38.x86_64                         11/28 
  Verifying        : libxml2-2.10.3-3.fc38.x86_64                         12/28 
  Verifying        : man-pages-6.04-1.fc38.noarch                         13/28 
  Verifying        : man-pages-6.03-3.fc38.noarch                         14/28 
  Verifying        : mesa-dri-drivers-23.0.2-2.fc38.x86_64                15/28 
  Verifying        : mesa-dri-drivers-23.0.1-1.fc38.x86_64                16/28 
  Verifying        : mesa-filesystem-23.0.2-2.fc38.x86_64                 17/28 
  Verifying        : mesa-filesystem-23.0.1-1.fc38.x86_64                 18/28 
  Verifying        : mesa-libglapi-23.0.2-2.fc38.x86_64                   19/28 
  Verifying        : mesa-libglapi-23.0.1-1.fc38.x86_64                   20/28 
  Verifying        : mesa-va-drivers-23.0.2-2.fc38.x86_64                 21/28 
  Verifying        : mesa-va-drivers-23.0.1-1.fc38.x86_64                 22/28 
  Verifying        : mesa-vulkan-drivers-23.0.2-2.fc38.x86_64             23/28 
  Verifying        : mesa-vulkan-drivers-23.0.1-1.fc38.x86_64             24/28 
  Verifying        : openssh-9.0p1-15.fc38.x86_64                         25/28 
  Verifying        : openssh-9.0p1-14.fc38.1.x86_64                       26/28 
  Verifying        : openssh-clients-9.0p1-15.fc38.x86_64                 27/28 
  Verifying        : openssh-clients-9.0p1-14.fc38.1.x86_64               28/28 

Upgraded:
  avahi-0.8-22.fc38.x86_64             avahi-libs-0.8-22.fc38.x86_64           
  hwdata-0.369-1.fc38.noarch           libwayland-client-1.22.0-1.fc38.x86_64  
  libxml2-2.10.4-1.fc38.x86_64         llvm-libs-16.0.0-2.fc38.x86_64          
  man-pages-6.04-1.fc38.noarch         mesa-dri-drivers-23.0.2-2.fc38.x86_64   
  mesa-filesystem-23.0.2-2.fc38.x86_64 mesa-libglapi-23.0.2-2.fc38.x86_64      
  mesa-va-drivers-23.0.2-2.fc38.x86_64 mesa-vulkan-drivers-23.0.2-2.fc38.x86_64
  openssh-9.0p1-15.fc38.x86_64         openssh-clients-9.0p1-15.fc38.x86_64    

Complete!


Expected Results:  
Ideally no errors or serious related warnings.

I don't think this was happening when Fedora 37 was released or even last year?
Though it is possible it could also be related to changes in fedora-toolbox content?

This also happens with fedora-toolbox:37,
but it seems fedora-toolbox:36 includes systemd pre-installed.

Comment 1 Jens Petersen 2023-04-20 12:15:57 UTC
I think this can also be reproduced directly with `podman run -it --rm fedora-toolbox dnf install systemd`.

Comment 2 Jens Petersen 2023-04-20 15:28:50 UTC
(Also I think originally mentioned Silverblue on #fedora-devel, but the above dnf results are from a F38 Workstation VM.)

Comment 3 Zbigniew Jędrzejewski-Szmek 2023-04-20 15:40:13 UTC
(In reply to Jens Petersen from comment #1)
> I think this can also be reproduced directly with `podman run -it --rm
> fedora-toolbox dnf install systemd`.

This doesn't reproduce here. But it works reproduces fine with the toolbox.

Those error messages are from 'systemd-tmpfiles --create 'which is executed by
the %transfiletriggerin scriptlet of systemd. This is easy enough to reproduce
manually:

$ sudo systemd-tmpfiles --create
fchownat() of /run/systemd/sessions failed: Operation not permitted
fchownat() of /run/systemd/users failed: Operation not permitted
fchownat() of /var/lib/systemd/coredump failed: Read-only file system
fchownat() of /tmp failed: Operation not permitted
Setting access ACL "u::rwx,g::r-x,g:adm:r-x,g:wheel:r-x,g:4294967295:r-x,g:4294967295:r-x,m::r-x,o::r-x" on /var/log/journal failed: Read-only file system
Failed to re-open '/var/log/journal': Operation not permitted
fchownat() of /var/log/journal failed: Read-only file system
Setting access ACL "u::rwx,g::r-x,g:adm:r-x,g:wheel:r-x,g:4294967295:r-x,g:4294967295:r-x,m::r-x,o::r-x" on /var/log/journal/3a9d668b4db749398a4a5e78a03bffa5 failed: Read-only file system
Failed to re-open '/var/log/journal/3a9d668b4db749398a4a5e78a03bffa5': Operation not permitted
fchownat() of /var/log/journal/3a9d668b4db749398a4a5e78a03bffa5 failed: Read-only file system
Failed to re-open '/var/log/journal/remote': Operation not permitted
fchownat() of /dev/snd/seq failed: Operation not permitted
fchownat() of /dev/snd/timer failed: Operation not permitted
fchownat() of /dev/loop-control failed: Operation not permitted
fchownat() of /dev/kvm failed: Operation not permitted
fchownat() of /dev/vhost-net failed: Operation not permitted
fchownat() of /dev/vhost-vsock failed: Operation not permitted
Setting access ACL "u::rw-,g::r-x,g:adm:r--,g:wheel:r--,g:4294967295:r--,g:4294967295:r--,m::r--,o::---" on /var/log/journal/3a9d668b4db749398a4a5e78a03bffa5/system.journal failed: Read-only file system
fchownat() of /var/log/journal/3a9d668b4db749398a4a5e78a03bffa5/system.journal failed: Read-only file system
fchownat() of /sys/kernel/security/tpm0/binary_bios_measurements failed: Operation not permitted
fchownat() of /sys/kernel/security/ima/binary_runtime_measurements failed: Operation not permitted

In the container:
$ ls -ld /run/systemd/sessions
drwxr-xr-x 2 nobody nobody 80 Apr 20 16:38 /run/systemd/sessions
$ ls -l  /sys/kernel/security/ima/binary_runtime_measurements
-r--r----- 1 nobody nobody 0 Apr 18 15:01 /sys/kernel/security/ima/binary_runtime_measurements

So this is a result of confusion about what the container is. It has parts of the host filesystem mounted into the namespace, so it's like a chroot, but it based on a real container and systemd detects it as such.

$ systemd-detect-virt 
podman

$ strace -e newfstatat systemd-detect-virt --chroot
newfstatat(AT_FDCWD, "/proc/1/root", 0x7ffeca036830, 0) = -1 EACCES (Permission denied)
Failed to check for chroot() environment: Permission denied

I'm not sure what to do here and how this hybrid design is supposed to work. I don't think there's anything wrong with what systemd is doing: we're told to execute some standard configuration by the scriptlet and config files in /etc, and we cannot do it because some file have unexpected ownership and permissions are denied. And in general, there are countless scriptlets that touch paths in the fs and don't expect them to be owned by nobody.

One option would be for toolbox to inject an override to for systemd-tmpfiles to not touch certain paths. This override would have to be match the paths that are mounted in from the host.

I'll reassign this to toolbox for comments.

Comment 4 Debarshi Ray 2023-04-27 17:52:46 UTC
We had hit something similar, but not exactly the same, before with paths marked with %attr(...) in RPMs.  eg., the filesystem RPM has `ghost %attr(555,root,root) /proc`.  See:
https://github.com/containers/toolbox/issues/643
https://bugzilla.redhat.com/show_bug.cgi?id=1830830

That was handled by placing a /usr/lib/rpm/macros.d/macros.toolbox with `%_netsharedpath` containing all such paths inside the Toolbx containers:
https://github.com/containers/toolbox/commit/7542f5fc867b57bf3dc67bbae02cc09ccc0b5df2

That won't help in this case.

Comment 5 Debarshi Ray 2023-04-27 18:44:28 UTC
Thanks for digging into this, Zbigniew!

(In reply to Zbigniew Jędrzejewski-Szmek from comment #3)
> 
> One option would be for toolbox to inject an override to for
> systemd-tmpfiles to not touch certain paths. This override would have to be
> match the paths that are mounted in from the host.

This sounds reasonable to me.  Since Toolbx does the bind mounts, it should also handle the overrides, so that everything can be tested and updated without having to co-ordinate it across multiple Git repositorie and downstream packages.

However, I don't know how the overrides will exactly work?  Toolbx will place some empty files in /run/tmpfiles.d that match the names of the files in /usr/lib/tmpfiles.d ?

Comment 6 Zbigniew Jędrzejewski-Szmek 2023-04-28 08:28:13 UTC
Yes:
/run/tmpfiles.d/systemd.conf
/run/tmpfiles.d/static-nodes-permissions.conf
/run/tmpfiles.d/static-nodes.conf

Comment 7 Debarshi Ray 2023-05-03 14:17:52 UTC
(In reply to Zbigniew Jędrzejewski-Szmek from comment #6)
> Yes:
> /run/tmpfiles.d/systemd.conf
> /run/tmpfiles.d/static-nodes-permissions.conf
> /run/tmpfiles.d/static-nodes.conf

Thanks for that list, Zbigniew!

I see that tmpfiles.d/systemd.conf.in contains a lot of things.  As far as systemd is concerned, Toolbx only bind mounts these locations from the host:
* /dev
* /run/systemd/journal
* /run/systemd/resolve
* /run/systemd/sessions
* /run/systemd/system
* /run/systemd/users
* /var/lib/systemd/coredump
* /var/log/journal

... to have working coredumpctl, journalctl, logging, DNS resolution, and running non-nested Wayland display servers.

I am wondering if it will be a good idea to override the whole of systemd.conf.  Maybe, Toolbx should bind mount all those other locations too?  Or maybe, Toolbx should filter out the bind mounted locations from the host's systemd.conf at run-time and install this filtered version as an override inside the container?

Everything in tmpfiles.d/static-nodes-permissions.conf.in comes from /dev which is bind mounted from the host.  So that's fine.

I see that static-nodes.conf is supposed to be generated by kmod-static-nodes.service but I don't have it on my Fedora host.  So, I will try to assume that it's safe to suppress it.  :)

Comment 8 Debarshi Ray 2023-05-03 14:24:48 UTC
(In reply to Zbigniew Jędrzejewski-Szmek from comment #6)
> Yes:
> /run/tmpfiles.d/systemd.conf
> /run/tmpfiles.d/static-nodes-permissions.conf
> /run/tmpfiles.d/static-nodes.conf

Also, since these filenames seem to be defined in the upstream systemd sources, I suppose I can rely on their names on other distributions, even those outside the Fedora universe, right?

Comment 9 Zbigniew Jędrzejewski-Szmek 2023-05-04 08:43:35 UTC
> since these filenames seem to be defined in the upstream systemd sources, I suppose I can rely on their names on other distributions, even those outside the Fedora universe, right?

Yes, those names are defined upstream and something that we very much try not to change,
because that breaks overrides. Unfortunately, there's a strong force in the other direction
too: people ask for files to be broken up to make it easier to override pieces. So
occasionally the names (or at least the placement of some lines in the config) do change.
Any such change happens upstream at a release boundary, with notices in NEWS.

> I see that static-nodes.conf is supposed to be generated by kmod-static-nodes.service but I don't have it on my Fedora host.

It is in /run/tmpfiles.d/static-nodes.conf. But all those files are in /dev,
so it would have to be suppressed too.

> I am wondering if it will be a good idea to override the whole of systemd.conf.  Maybe, Toolbx should bind mount all those other locations too?  Or maybe, Toolbx should filter out the bind mounted locations from the host's systemd.conf at run-time and install this filtered version as an override inside the container?

All the locations defined in systemd.conf are closely tied to the functioning of
the host's pid1, so I don't think the container should ever create those files.
Toolbx should either bind-mount the subset that makes sense for the container,
and suppress the file as a whole.

Comment 10 Debarshi Ray 2023-05-04 14:34:07 UTC
Okay, understood.  Thanks for taking the time to explain all that in detail.

Comment 11 Aoife Moloney 2024-05-07 16:08:02 UTC
This message is a reminder that Fedora Linux 38 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora Linux 38 on 2024-05-21.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
'version' of '38'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, change the 'version' 
to a later Fedora Linux version. Note that the version field may be hidden.
Click the "Show advanced fields" button if you do not see it.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora Linux 38 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora Linux, you are encouraged to change the 'version' to a later version
prior to this bug being closed.


Note You need to log in before you can comment on or make changes to this bug.