Description of problem: I've pulled the rawhide image and tried doing a dnf --update: FROM fedora:rawhide # Update the container RUN true \ && dnf update -y --refresh \ && true CMD true \ && bash -i \ && true When executing this under buildah as a regular user, I get the following: $ buildah bud -f bugreports/fedora_filesystem --tag fedora_filesystem:rawhide . STEP 1: FROM fedora:rawhide STEP 2: RUN true && dnf update -y --refresh && true Fedora - Modular Rawhide - Developmental packages for the next Fedora release 147 kB/s | 198 kB 00:01 Fedora - Rawhide - Developmental packages for the next Fedora release 4.0 MB/s | 56 MB 00:13 Dependencies resolved. ======================================================================================================================================= Package Architecture Version Repository Size ======================================================================================================================================= Upgrading: <snip> filesystem x86_64 3.11-1.fc31 rawhide 1.1 M <snip> Upgrading : filesystem-3.11-1.fc31.x86_64 59/144 Error unpacking rpm package filesystem-3.11-1.fc31.x86_64 error: unpacking of archive failed on file /proc: cpio: chown Installing : xkeyboard-config-2.24-5.fc30.noarch 60/144 error: filesystem-3.11-1.fc31.x86_64: install failed <snip> Failed: filesystem-3.11-1.fc31.x86_64 Error: Transaction failed Version-Release number of selected component (if applicable): filesystem-3.11-1.fc31.x86_64 How reproducible: Very, see above. Steps to Reproduce: 1. Write above Dockerfile to disk. 2. Run buildah bud -f ThisBZDockerFile --tag fedora_filesystem:rawhide . 3. Notice that filesystem fails to update. Actual results: filesystem fails to update because of an error unpacking file Expected results: Fedora rawhide should be able to update. Additional info: See attached error.log for a complete text. This is using stock Rawhide
Created attachment 1566144 [details] Full output of dnf update on a Fedora rawhide container
I do not think bug is in the filesystem package. Build of image passed for me without any problem. What is version of fedora on host? Could you provide output of following commands from your machine ? sh# ls -ld /sys/ sh# rpm -V filesystem
Sure, happy to: -- Host system #1 -- $ lsb_release -a LSB Version: :core-4.1-amd64:core-4.1-noarch Distributor ID: Fedora Description: Fedora release 30 (Thirty) Release: 30 Codename: Thirty $ ls -ld /sys/ dr-xr-xr-x. 13 root root 0 Apr 30 18:26 /sys/ $ rpm -V filesystem $ rpm -qa | grep ^filesystem filesystem-3.10-1.fc30.x86_64 $ sudo dnf update --refresh -y Fedora Modular 30 - x86_64 2.6 kB/s | 15 kB 00:05 Fedora Modular 30 - x86_64 - Updates 31 kB/s | 15 kB 00:00 Fedora 30 - x86_64 - Updates 67 kB/s | 17 kB 00:00 Fedora 30 - x86_64 52 kB/s | 15 kB 00:00 google-chrome 7.9 kB/s | 1.3 kB 00:00 RCM Tools for Fedora 30 (RPMs) 3.6 kB/s | 3.8 kB 00:01 RPM Fusion for Fedora 30 - Free - Updates 5.9 kB/s | 3.6 kB 00:00 RPM Fusion for Fedora 30 - Free 13 kB/s | 3.2 kB 00:00 RPM Fusion for Fedora 30 - Nonfree - Updates 4.4 kB/s | 3.7 kB 00:00 RPM Fusion for Fedora 30 - Nonfree 5.3 kB/s | 3.2 kB 00:00 Dependencies resolved. Nothing to do. Complete! $ buildah images docker.io/library/fedora rawhide b994d52a3692 Apr 24, 2019 17:21 330 MB -- Host system #2 -- $ lsb_release -a LSB Version: :core-4.1-amd64:core-4.1-noarch Distributor ID: Fedora Description: Fedora release 29 (Twenty Nine) Release: 29 Codename: TwentyNine $ ls -ld /sys/ dr-xr-xr-x. 13 root root 0 Jan 29 12:00 /sys/ $ rpm -V filesystem $ rpm -qa | grep ^filesystem filesystem-3.9-2.fc29.x86_64 $ sudo dnf update --refresh -y Copr repo for master owned by @pki 11 kB/s | 3.5 kB 00:00 ccutil for Fedora 29 32 kB/s | 2.9 kB 00:00 Fedora Modular 29 - x86_64 49 kB/s | 18 kB 00:00 Fedora Modular 29 - x86_64 - Updates 98 kB/s | 15 kB 00:00 Fedora 29 - x86_64 - Updates 29 kB/s | 16 kB 00:00 Fedora 29 - x86_64 51 kB/s | 18 kB 00:00 RCM Tools for Fedora 29 (RPMs) 42 kB/s | 3.8 kB 00:00 RPM Fusion for Fedora 29 - Free - Updates 18 kB/s | 3.6 kB 00:00 RPM Fusion for Fedora 29 - Free 4.4 kB/s | 3.2 kB 00:00 RPM Fusion for Fedora 29 - Nonfree - Updates 8.6 kB/s | 3.7 kB 00:00 RPM Fusion for Fedora 29 - Nonfree 29 kB/s | 3.2 kB 00:00 Dependencies resolved. Nothing to do. Complete! $ buildah images docker.io/library/fedora rawhide eff97b05aaa6 Mar 11, 2019 20:20 311 MB
The issue is in buildah when running in unprivileged mode Owner and group for /proc is nobody inside container for unprivileged user I prepared simpler and deterministic reproducer [test@hp-dl180g6-01 ~]$ cat Dockerfile FROM docker.io/fedora@sha256:ac62b7859a2e8f1339c6ec4bc4bbafcd352ffc385d632c9819e6f8942ff91416 # Update the container RUN true \ && capsh --print \ && ls -ld /proc/ \ && dnf update -y filesystem [test@hp-dl180g6-01 ~]$ ls -ld /proc/ dr-xr-xr-x. 216 root root 0 May 8 18:07 /proc/ [test@hp-dl180g6-01 ~]$ buildah bud --tag fedora_filesystem:rawhide . STEP 1: FROM docker.io/fedora@sha256:ac62b7859a2e8f1339c6ec4bc4bbafcd352ffc385d632c9819e6f8942ff91416 STEP 2: RUN true && capsh --print && ls -ld /proc/ && dnf update -y filesystem Current: = cap_chown,cap_dac_override,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_net_bind_service,cap_sys_chroot,cap_mknod,cap_audit_write,cap_setfcap+eip Bounding set =cap_chown,cap_dac_override,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_net_bind_service,cap_sys_chroot,cap_mknod,cap_audit_write,cap_setfcap Ambient set =cap_chown,cap_dac_override,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_net_bind_service,cap_sys_chroot,cap_mknod,cap_audit_write,cap_setfcap Securebits: 00/0x0/1'b0 secure-noroot: no (unlocked) secure-no-suid-fixup: no (unlocked) secure-keep-caps: no (unlocked) secure-no-ambient-raise: no (unlocked) uid=0(root) gid=0(root) groups= dr-xr-xr-x. 222 nobody nobody 0 May 9 19:32 /proc/ Fedora - Modular Rawhide - Developmental packages for the next Fedora rel 42 kB/s | 198 kB 00:04 Fedora - Rawhide - Developmental packages for the next Fedora release 1.1 MB/s | 56 MB 00:50 Dependencies resolved. ========================================================================================================== Package Architecture Version Repository Size ========================================================================================================== Upgrading: filesystem x86_64 3.11-1.fc31 rawhide 1.1 M Transaction Summary ========================================================================================================== Upgrade 1 Package Total download size: 1.1 M Downloading Packages: filesystem-3.11-1.fc31.x86_64.rpm 378 kB/s | 1.1 MB 00:02 ---------------------------------------------------------------------------------------------------------- Total 356 kB/s | 1.1 MB 00:03 Running transaction check Transaction check succeeded. Running transaction test Transaction test succeeded. Running transaction Running scriptlet: filesystem-3.11-1.fc31.x86_64 1/1 Preparing : 1/1 Upgrading : filesystem-3.11-1.fc31.x86_64 1/2 Error unpacking rpm package filesystem-3.11-1.fc31.x86_64 error: unpacking of archive failed on file /proc: cpio: chown Cleanup : filesystem-3.10-1.fc30.x86_64 2/2 error: filesystem-3.11-1.fc31.x86_64: install failed Running scriptlet: filesystem-3.10-1.fc30.x86_64 2/2 Verifying : filesystem-3.11-1.fc31.x86_64 1/2 Verifying : filesystem-3.10-1.fc30.x86_64 2/2 Failed: filesystem-3.11-1.fc31.x86_64 Error: Transaction failed error building at step {Env:[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin DISTTAG=f31container FGC=f31 FBR=f31] Command:run Args:[true && capsh --print && ls -ld /proc/ && dnf update -y filesystem] Flags:[] Attrs:map[] Message:RUN true && capsh --print && ls -ld /proc/ && dnf update -y filesystem Original:RUN true && capsh --print && ls -ld /proc/ && dnf update -y filesystem}: error while running runtime: exit status 1 ERRO[0106] exit status 1
Workaround is to run "buildah bud" as a root or to use newer image for rawhide (registry.fedoraproject.org/fedora:rawhide) The filesystem package is up to date there
I would guess it is caused by fuse-overlayfs vs overlay for /
I can confirm that running buildah as root is a workaround for this problem from my end. Thanks!
buildah-1.8.2-1.gite23314b.fc29 has been submitted as an update to Fedora 29. https://bodhi.fedoraproject.org/updates/FEDORA-2019-204dcd7630
buildah-1.8.2-1.gite23314b.fc30 has been submitted as an update to Fedora 30. https://bodhi.fedoraproject.org/updates/FEDORA-2019-e4d7ff8447
Giuseppe, do you think this is an error in fuse-overlay?
this is more of a problem in the rpm package that is trying to chown /proc, which is not handled by fuse-overlayfs. That is not permitted for an unprivileged user.
buildah-1.8.2-1.gite23314b.fc30 has been pushed to the Fedora 30 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-e4d7ff8447
buildah-1.8.2-1.gite23314b.fc29 has been pushed to the Fedora 29 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-204dcd7630
(In reply to Giuseppe Scrivano from comment #11) > this is more of a problem in the rpm package that is trying to chown /proc, > which is not handled by fuse-overlayfs. That is not permitted for an > unprivileged user. It works well when running as root so it cannot be problem in rpm. The problem is that UID/GID is not expected in unprivileged mode and moreover it si not allowed to fix that in case of upgrade.
Lukas what UID/GID is it using? Is it a huge UID?
(In reply to Daniel Walsh from comment #15) > Lukas what UID/GID is it using? Is it a huge UID? You can see it in the comment#4. It's nobody user which is usually high And here is output from following Dockerfile FROM docker.io/fedora@sha256:ac62b7859a2e8f1339c6ec4bc4bbafcd352ffc385d632c9819e6f8942ff91416 # Update the container RUN true \ && capsh --print \ && ls -lnd /* \ && rpm -V filesystem \ && dnf update -y filesystem sh$ buildah bud --tag fedora_filesystem:rawhide . STEP 1: FROM docker.io/fedora@sha256:ac62b7859a2e8f1339c6ec4bc4bbafcd352ffc385d632c9819e6f8942ff91416 STEP 2: RUN true && capsh --print && ls -lnd /* && rpm -V filesystem && dnf update -y filesystem Current: = cap_chown,cap_dac_override,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_net_bind_service,cap_sys_chroot,cap_mknod,cap_audit_write,cap_setfcap+eip Bounding set =cap_chown,cap_dac_override,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_net_bind_service,cap_sys_chroot,cap_mknod,cap_audit_write,cap_setfcap Ambient set =cap_chown,cap_dac_override,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_net_bind_service,cap_sys_chroot,cap_mknod,cap_audit_write,cap_setfcap Securebits: 00/0x0/1'b0 secure-noroot: no (unlocked) secure-no-suid-fixup: no (unlocked) secure-keep-caps: no (unlocked) secure-no-ambient-raise: no (unlocked) uid=0(root) gid=0(root) groups= lrwxrwxrwx. 1 0 0 7 Feb 11 13:47 /bin -> usr/bin dr-xr-xr-x. 2 0 0 6 Feb 11 13:47 /boot drwxr-xr-x. 5 0 0 360 May 15 11:29 /dev drwxr-xr-x. 3 0 0 4096 Apr 22 11:39 /etc drwxr-xr-x. 2 0 0 6 Feb 11 13:47 /home lrwxrwxrwx. 1 0 0 7 Feb 11 13:47 /lib -> usr/lib lrwxrwxrwx. 1 0 0 9 Feb 11 13:47 /lib64 -> usr/lib64 drwx------. 2 0 0 6 Apr 22 11:38 /lost+found drwxr-xr-x. 2 0 0 6 Feb 11 13:47 /media drwxr-xr-x. 2 0 0 6 Feb 11 13:47 /mnt drwxr-xr-x. 2 0 0 6 Feb 11 13:47 /opt dr-xr-xr-x. 171 65534 65534 0 May 15 11:29 /proc dr-xr-x---. 2 0 0 162 Apr 22 11:39 /root drwxr-xr-x. 2 0 0 174 Apr 22 11:39 /run lrwxrwxrwx. 1 0 0 8 Feb 11 13:47 /sbin -> usr/sbin drwxr-xr-x. 2 0 0 6 Feb 11 13:47 /srv dr-xr-xr-x. 13 65534 65534 0 May 15 01:33 /sys drwxrwxrwt. 2 0 0 32 Apr 22 11:39 /tmp drwxr-xr-x. 7 0 0 144 Apr 22 11:39 /usr drwxr-xr-x. 3 0 0 235 Apr 22 11:39 /var .M....... / .....UG.. /proc .....UG.. /sys error building at step {Env:[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin DISTTAG=f31container FGC=f31 FBR=f31] Command:run Args:[true && capsh --print && ls -lnd /* && rpm -V filesystem && dnf update -y filesystem] Flags:[] Attrs:map[] Message:RUN true && capsh --print && ls -lnd /* && rpm -V filesystem && dnf update -y filesystem Original:RUN true && capsh --print && ls -lnd /* && rpm -V filesystem && dnf update -y filesystem}: error while running runtime: exit status 1 ERRO[0000] exit status 1 So it is not just about /proc but also for /sys
I take it that this works if you run buildah bud as root but not in rootless mode.
What is attempting to install the filesystem package? In the default rawhide container, there is no file system package, and it looks like something is attempting to pull it in?
Dan, I'm not sure I follow: $ podman run -ti registry.fedoraproject.org/fedora:rawhide /bin/bash -c 'rpm -qa | grep filesystem' Trying to pull registry.fedoraproject.org/fedora:rawhide...Getting image source signatures Copying blob 05af05297975 done Copying config fc02a8623a done Writing manifest to image destination Storing signatures libreport-filesystem-2.10.0-3.fc31.noarch filesystem-3.11-1.fc31.x86_64 $ podman run -ti registry.hub.docker.com/library/fedora:rawhide /bin/bash -c 'rpm -qa | grep filesystem' Trying to pull registry.hub.docker.com/library/fedora:rawhide...Getting image source signatures Copying blob 5036ff40a678 skipped: already exists Copying config b994d52a36 done Writing manifest to image destination Storing signatures libreport-filesystem-2.10.0-1.fc30.noarch filesystem-3.10-1.fc30.x86_64 This isn't "attempting to install" -- this is a core package and I doubt Fedora could function without it. Description : The filesystem package is one of the basic packages that is installed : on a Linux system. Filesystem contains the basic directory layout : for a Linux operating system, including the correct permissions for : the directories. This includes a bunch of default filesystem directories such as everything in / (/root, /etc, ...) (now, skipping the update for this package is probably fine, but I'm more annoyed with it breaking dnf update as a result) And yes: $ podman run -ti registry.hub.docker.com/library/fedora:rawhide dnf update --refresh -y ^ fails when run as a regular user But: $ sudo podman run -ti registry.hub.docker.com/library/fedora:rawhide dnf update --refresh -y ^ works when run as root via sudo. And: $ sudo docker run -ti registry.hub.docker.com/library/fedora:rawhide dnf update --refresh -y ^ works as well. I'm too lazy to set up docker as not-root, I'll leave that as an exercise for the reader. The lazy solution would be to poke $SOMEBODY to update the Dockerhub image, which I'm perfectly fine with. Or have podman/buildah detect that you're trying to load an unqualified "fedora:<version>" and pull that from Fedora's registry instead (which'd require more discussion probably).
/proc is owned by root in the initial namespace, so a chown on /proc in a rootless container will never succeed when it is mounted. You can workaround it by mounting something else on /proc (hopefully yum/rpm won't need it), but the correct fix would be to not attempt to chown it. Can cpio chown errors be ignored?
(In reply to Giuseppe Scrivano from comment #20) > /proc is owned by root in the initial namespace, so a chown on /proc in a > rootless container will never succeed when it is mounted. > > You can workaround it by mounting something else on /proc (hopefully yum/rpm > won't need it), but the correct fix would be to not attempt to chown it. > Can cpio chown errors be ignored? You cannot ignore chown errors in rpm. You want to fix owner/permissions in chase of spec file change. The problem is that /proc and /sys use nobody user and not root. But that probably cannot be changed for rootless mode.
Well the /proc and /sys are actually owned by real root, but in side of the user namespace, real root is not mapped, so files owned by UID=0 are shown as NOBODY. The rawhide image comes without a filesytem package inside of it, so these tools are attempting to install it. sudo podman run fedora:rawhide rpm -q filesystem [sudo] password for dwalsh: filesystem-3.10-1.fc30.x86_64 The problem is the rawhide fedora container image is screwed up right now. sudo podman run fedora:rawhide rpm -qf /etc/fedora-release /proc fedora-release-common-31-0.5.noarch filesystem-3.10-1.fc30.x86_64 The release is 31, but the filesystem package is 30, which is triggering the filesystem package to require an update. Probably most releases this package never gets updated, so no one ever sees this issue. Once fedora:rawhide actually has packages built for rawhide, the issue should go away.
Who is in charge of creating the base images? How do I reassign this to tell Fedora to fix the base image.
(In reply to Daniel Walsh from comment #23) > Who is in charge of creating the base images? How do I reassign this to > tell Fedora to fix the base image. That is just workaround and the same situation will happen next time in case of new rawhide koji build for filesystem package.
Clement, could you update rawhide image in docker hub ?
(In reply to Daniel Walsh from comment #22) > Well the /proc and /sys are actually owned by real root, but in side of the > user namespace, real root is not mapped, so files owned by UID=0 are shown > as NOBODY. > > The rawhide image comes without a filesytem package inside of it, so these > tools are attempting to install it. > The rawhide image comes *WITH* the filesystem package. sh# docker run -ti --rm docker.io/fedora:rawhide rpm -q filesystem filesystem-3.10-1.fc30.x86_64 And the filesystem pacakge is pulled in as a dependency of bash/gawk sh# docker run -ti --rm docker.io/fedora:rawhide rpm -e filesystem error: Failed dependencies: filesystem >= 3 is needed by (installed) bash-5.0.2-1.fc30.x86_64 filesystem >= 3 is needed by (installed) gawk-4.2.1-6.fc31.x86_64
Perhaps a workaround, but since this can not be fixed, and is fairly unusual, it is the best we can do. If you need to update a package that attempts to modify files owned by root within a usernamespaced container it will fail. Luckily the only exposed files by default are /proc and /sys.
Filesystem package within the rawhide container image should be from f31 not f30. All packages in that image should be checked to make sure they are update.
This is just a normal rawhide process. Until the F31 mass rebuild (2019-07-24 according to the schedule [0]), only things which have been explicitly updated (such as filesystem) will have fc31. Everything else which is the same since F30 branched (and which haven't been updated in rawhide) will have the fc30 tag. The bug is due to the mapping of root->NOBODY. Shouldn't root be mapped to "root" inside the container, with changes not being propagated to outside the container? I'm sure you'll tell me this isn't possible due to using bind mounting or something, but that satisfies the principle of least surprise: I have root==uid=0,gid=0 inside the container, so I should be able to chown darn near everything inside the container. At the very least, even if chown doesn't work, the owner of /proc and /sys *inside* the container should appear as being root. But I don't know what's required to get there or what else that'd break. My 2c. [0]: https://fedoraproject.org/wiki/Releases/31/Schedule
Alex, I understand, I have worked on Fedora since the fedora core 1. But the filesystem package is available now, and has not been built in the rawhide image yet. I am not arguing that this package should be allowed to be updated and is working correctly. I am explaining that running container runtime tools like podman and buildah as non root will not work in all circumstances. We have a long list of them. The one we are hitting here is a root file inside of a user namespace without UID=0 being mounted inside of the container. When you run rootless containers usually your UID is root inside of the container or the first UID listed in /etc/subuid. But UID=0 on the host is not mapped into the container, and even it it was you would not be allowed to modify the root owned files like the /proc and /sys directories. User Namespace says that any UID on the file systme that is not mapped into th user namespace is treated as the NOBODY user and is only accessible via WORLD permissions.
Bottom line we can not fix this. If you need to install/update the filesystem package you have to run podman/buildah as root.
buildah-1.8.2-1.gite23314b.fc30 has been pushed to the Fedora 30 stable repository. If problems still persist, please make note of it in this bug report.
buildah-1.8.2-1.gite23314b.fc29 has been pushed to the Fedora 29 stable repository. If problems still persist, please make note of it in this bug report.
*** Bug 1548403 has been marked as a duplicate of this bug. ***
(In reply to Daniel Walsh from comment #31) > Bottom line we can not fix this. > If you need to install/update the filesystem package you have to run > podman/buildah as root. Is this still valid statement? If not, do we have more info?
Pavel it is always going to be a valid statement. A non root process running as your user is not going to be able to chown a file system /proc and /sys owned by root. There is always going to be some things we can not do in rootless mode, since we are under the constraints of not being root. here is a whole page of things we can not do in rootless mode. https://github.com/containers/podman/blob/master/rootless.md Now maybe in the future the kernel will support a rootless user mounting the /proc and /sys file system in their user namespace, and this will be owned by root of the user namespace, but I would not bet on this happening soon.
At least the initial problem has been fixed. $ podman run fedora:rawhide rpm -q filesystem filesystem-3.14-2.fc32.x86_64
Thanks for the explanation! I was just confused by the "CANTFIX => ERRATA" status change. Mostly triggered by buildah bodhi update. > $ podman run fedora:rawhide rpm -q filesystem > filesystem-3.14-2.fc32.x86_64 Yeah, I think the current devel discussion is happening because that one is already obsolete, there should be fc33 package in Rawhide.