This can be easily reproduced on Fedora 33 x86_64 using mock: $ mock -r fedora-rawhide-armhfp --shell .... Observed by @jkadlcik, while running the Copr testsuite. Steps to reproduce without mock /on F33 x86_64/ #> # quite some free space .... #> df -h /var/tmp/ Filesystem Size Used Avail Use% Mounted on /dev/mapper/luks-.... 477G 374G 103G 79% / #> # install bootstrap chroot #> dnf -y --installroot /var/tmp/main-root --releasever 34 --forcearch armv7hl install dnf --setopt=tsflags=nocontexts --disablerepo='*' --enablerepo fedora --enablerepo updates #> # Necessary changes to make dnf in chroot work # rm -f /var/tmp/main-root-rawhide/etc/resolv.conf # cp /etc/resolv.conf /var/tmp/main-root-rawhide/etc/resolv.conf # mount --bind /dev/urandom /var/tmp/main-root-rawhide/dev/urandom # chroot /var/tmp/main-root/ bash-5.0# /usr/bin/dnf --installroot /sub-root --releasever 34 --forcearch armv7hl install filesystem --setopt=tsflags=nocontexts --disablerepo='*' --enablerepo fedora --enablerepo updates Failed to set locale, defaulting to C.UTF-8 Fedora 34 - armhfp - Updates 2.3 MB/s | 69 MB 00:30 Fedora 34 - armhfp 324 kB/s | 1.6 MB 00:05 Last metadata expiration check: 0:01:34 ago on Fri Nov 6 13:27:44 2020. Dependencies resolved. ============================================================================================================================================================================================================================================ Package Architecture Version Repository Size ============================================================================================================================================================================================================================================ Installing: filesystem armv7hl 3.14-4.fc34 updates 1.1 M Installing dependencies: basesystem noarch 11-10.fc33 updates 6.8 k bash armv7hl 5.0.17-2.fc33 updates 1.6 M fedora-gpg-keys noarch 34-0.8 updates 107 k fedora-release noarch 34-0.9 updates 12 k fedora-release-common noarch 34-0.9 updates 21 k fedora-release-identity-basic noarch 34-0.9 updates 13 k fedora-repos noarch 34-0.8 updates 11 k fedora-repos-rawhide noarch 34-0.8 updates 10 k glibc armv7hl 2.32.9000-12.fc34 updates 3.1 M glibc-common armv7hl 2.32.9000-12.fc34 updates 1.8 M glibc-minimal-langpack armv7hl 2.32.9000-12.fc34 updates 93 k libgcc armv7hl 10.2.1-6.fc34 updates 102 k ncurses-base noarch 6.2-3.20200222.fc33 updates 60 k ncurses-libs armv7hl 6.2-3.20200222.fc33 updates 285 k setup noarch 2.13.7-2.fc33 updates 142 k tzdata noarch 2020d-1.fc34 updates 430 k Transaction Summary ============================================================================================================================================================================================================================================ Install 17 Packages Total download size: 8.9 M Installed size: 31 M Is this ok [y/N]: y Downloading Packages: (1/17): basesystem-11-10.fc33.noarch.rpm 34 kB/s | 6.8 kB 00:00 (2/17): fedora-release-34-0.9.noarch.rpm 315 kB/s | 12 kB 00:00 (3/17): fedora-release-common-34-0.9.noarch.rpm 492 kB/s | 21 kB 00:00 (4/17): fedora-gpg-keys-34-0.8.noarch.rpm 363 kB/s | 107 kB 00:00 (5/17): fedora-release-identity-basic-34-0.9.noarch.rpm 468 kB/s | 13 kB 00:00 (6/17): fedora-repos-34-0.8.noarch.rpm 400 kB/s | 11 kB 00:00 (7/17): fedora-repos-rawhide-34-0.8.noarch.rpm 342 kB/s | 10 kB 00:00 (8/17): filesystem-3.14-4.fc34.armv7hl.rpm 1.2 MB/s | 1.1 MB 00:00 (9/17): bash-5.0.17-2.fc33.armv7hl.rpm 1.3 MB/s | 1.6 MB 00:01 (10/17): glibc-minimal-langpack-2.32.9000-12.fc34.armv7hl.rpm 1.1 MB/s | 93 kB 00:00 (11/17): libgcc-10.2.1-6.fc34.armv7hl.rpm 1.1 MB/s | 102 kB 00:00 (12/17): ncurses-base-6.2-3.20200222.fc33.noarch.rpm 421 kB/s | 60 kB 00:00 (13/17): ncurses-libs-6.2-3.20200222.fc33.armv7hl.rpm 1.1 MB/s | 285 kB 00:00 (14/17): setup-2.13.7-2.fc33.noarch.rpm 1.0 MB/s | 142 kB 00:00 (15/17): tzdata-2020d-1.fc34.noarch.rpm 1.1 MB/s | 430 kB 00:00 (16/17): glibc-2.32.9000-12.fc34.armv7hl.rpm 1.3 MB/s | 3.1 MB 00:02 (17/17): glibc-common-2.32.9000-12.fc34.armv7hl.rpm 1.2 MB/s | 1.8 MB 00:01 -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Total 2.3 MB/s | 8.9 MB 00:03 warning: [fd 19]: Header V4 RSA/SHA256 Signature, key ID 45719a39: NOKEY Fedora 34 - armhfp - Updates 660 kB/s | 1.6 kB 00:00 Importing GPG key 0x45719A39: Userid : "Fedora (34) <fedora-34-primary>" Fingerprint: 8C5B A699 0BDB 26E1 9F2A 1A80 1161 AE69 4571 9A39 From : /etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-34-armhfp Is this ok [y/N]: y Key imported successfully Running transaction check Transaction check succeeded. Running transaction test The downloaded packages were saved in cache until the next successful transaction. You can remove cached packages by executing 'dnf clean packages'. Error: Transaction test error: installing package fedora-release-identity-basic-34-0.9.noarch needs 44KB more space on the / filesystem installing package tzdata-2020d-1.fc34.noarch needs 7MB more space on the / filesystem installing package ncurses-base-6.2-3.20200222.fc33.noarch needs 7MB more space on the / filesystem installing package fedora-gpg-keys-34-0.8.noarch needs 8MB more space on the / filesystem installing package libgcc-10.2.1-6.fc34.armv7hl needs 9MB more space on the / filesystem installing package fedora-release-34-0.9.noarch needs 9MB more space on the / filesystem installing package fedora-release-common-34-0.9.noarch needs 9MB more space on the / filesystem installing package fedora-repos-rawhide-34-0.8.noarch needs 9MB more space on the / filesystem installing package fedora-repos-34-0.8.noarch needs 9MB more space on the / filesystem installing package setup-2.13.7-2.fc33.noarch needs 10MB more space on the / filesystem installing package filesystem-3.14-4.fc34.armv7hl needs 14MB more space on the / filesystem installing package basesystem-11-10.fc33.noarch needs 14MB more space on the / filesystem installing package glibc-minimal-langpack-2.32.9000-12.fc34.armv7hl needs 15MB more space on the / filesystem installing package glibc-common-2.32.9000-12.fc34.armv7hl needs 23MB more space on the / filesystem installing package glibc-2.32.9000-12.fc34.armv7hl needs 39MB more space on the / filesystem installing package ncurses-libs-6.2-3.20200222.fc33.armv7hl needs 40MB more space on the / filesystem installing package bash-5.0.17-2.fc33.armv7hl needs 48MB more space on the / filesystem Error Summary ------------- Disk Requirements: At least 48MB more space needed on the / filesystem. bash-5.0#
FWIW, I'm not able to reproduce with the above instructions, in mock or otherwise (after correcting /var/tmp/main-root-rawhide/ vs /var/tmp/main-root/ inconsistencies in the instructions). So there seems to be some missing ingredient. But with the non-mock instructions I get this, which might be related... bash-5.0# df /usr/bin/df: cannot read table of mounted file systems: No such file or directory > This can be easily reproduced on Fedora 33 x86_64 using mock: > $ mock -r fedora-rawhide-armhfp --shell > ... Is that "mock -r fedora-rawhide-armhfp --shell" alone supposed to reproduce it, or does "..." mean some "business as usual" commands that might be relevant after all?
> after correcting /var/tmp/main-root-rawhide/ vs /var/tmp/main-root/ Yes, sorry. I tested this again onw on my F33, so I corrected the instructions: rpm -q rpm dnf rpm-4.16.0-1.fc33.x86_64 dnf-4.4.0-3.fc33.noarch Steps to reproduce: sudo su - dnf -y --installroot /var/tmp/main-root --releasever 34 --forcearch armv7hl install dnf --setopt=tsflags=nocontexts --disablerepo='*' --enablerepo fedora --enablerepo updates # the resolv.conf symlink is not created anymore by systemd, so no need to # remove it now cp /etc/resolv.conf /var/tmp/main-root/etc/resolv.conf touch /var/tmp/main-root/dev/urandom mount --bind /dev/urandom /var/tmp/main-root/dev/urandom chroot /var/tmp/main-root/ bash-5.0# /usr/bin/dnf -y --installroot /sub-root --releasever 34 --forcearch armv7hl install filesystem --setopt=tsflags=nocontexts --disablerepo='*' --enablerepo fedora --enablerepo updates ... Error: Transaction test error: installing package fedora-release-identity-basic-34-0.9.noarch needs 44KB more space on the / filesystem ... Error Summary ------------- Disk Requirements: At least 48MB more space needed on the / filesystem. > Is that "mock -r fedora-rawhide-armhfp --shell" alone supposed to reproduce it Yes, but I guess I should suggest 'mock -r fedora-rawhide-armhfp --scrub=all' first, and turn on (if explicitly disabled) the bootstrap. The mock process fails when it tries to install the build chroot from the (emulated) bootstrap chroot.
This seems rather unstable at best. I can now fairly reliably reproduce the first failure (outside mock), but what happens after than varies a lot. Sometimes the second attempt at the last command succeeds, sometimes it doesn't, and so on. When it fails, rpm on debug logging shows: D: computing file dispositions D: 0x0000fd00 4096 0 4062891 rotational:-1 / Which says 4096 blocksize, 0 available blocks and 4062891 available inodes on the / filesystem. Zero available blocks could be from read-only filesystem indicated by statvfs(), but other than that it's just whatever statvfs() returns, and rpm is right to stop when there's no space indicated (or the media is read-only, which is mapped to 0 avail by rpm) What does this forcearch-thing really do? That's where I would look at first.
Figured I can strace the thing from the outside. This is what that activity looks like when run from the host (with rpm -Uvv --ignorearch): stat("/var/lib/rpm", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 statfs("/", {f_type=EXT2_SUPER_MAGIC, f_bsize=4096, f_blocks=17931038, f_bfree=4107696, f_bavail=3186096, f_files=4587520, f_ffree=4078319, f_fsid={val=[3681299638, 2236116973]}, f_namelen=255, f_frsize=4096, f_flags=ST_VALID|ST_RELATIME}) = 0 stat("/", {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0 stat("/", {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0 write(2, "D: ", 3) = 3 write(2, "0x0000fd00 4096 3186096"..., 62) = 62 What happens in the qemu-static process is something quite different: statx(AT_FDCWD, "/var/lib/rpm", AT_STATX_SYNC_AS_STAT|AT_NO_AUTOMOUNT, STATX_BASIC_STATS, {stx_mask=STATX_BASIC_STATS|STATX_MNT_ID, stx_attributes=0, stx_mode=S_IFDIR|0755, stx_size=4096, ...}) = 0 mmap(NULL, 1052672, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f1de2bc6000 futex(0x7e7dc8, FUTEX_WAKE, 2147483647) = 1 access("/usr/qemu-arm/usr/lib/", F_OK) = -1 ENOENT (No such file or directory) statfs("/usr/lib/", {f_type=EXT2_SUPER_MAGIC, f_bsize=4096, f_blocks=17931038, f_bfree=3938041, f_bavail=3016441, f_files=4587520, f_ffree=4062853, f_fsid={val=[3681299638, 2236116973]}, f_namelen=255, f_frsize=4096, f_flags=ST_VALID|ST_RELATIME}) = 0 statx(AT_FDCWD, "/usr/lib/", AT_STATX_SYNC_AS_STAT|AT_NO_AUTOMOUNT, STATX_BASIC_STATS, {stx_mask=STATX_BASIC_STATS|STATX_MNT_ID, stx_attributes=0, stx_mode=S_IFDIR|0755, stx_size=4096, ...}) = 0 statx(AT_FDCWD, "/usr", AT_STATX_SYNC_AS_STAT|AT_SYMLINK_NOFOLLOW|AT_NO_AUTOMOUNT, STATX_BASIC_STATS, {stx_mask=STATX_BASIC_STATS|STATX_MNT_ID, stx_attributes=0, stx_mode=S_IFDIR|0755, stx_size=4096, ...}) = 0 statx(AT_FDCWD, "/usr/lib", AT_STATX_SYNC_AS_STAT|AT_SYMLINK_NOFOLLOW|AT_NO_AUTOMOUNT, STATX_BASIC_STATS, {stx_mask=STATX_BASIC_STATS|STATX_MNT_ID, stx_attributes=0, stx_mode=S_IFDIR|0755, stx_size=4096, ...}) = 0 statx(AT_FDCWD, "/usr", AT_STATX_SYNC_AS_STAT|AT_NO_AUTOMOUNT, STATX_BASIC_STATS, {stx_mask=STATX_BASIC_STATS|STATX_MNT_ID, stx_attributes=0, stx_mode=S_IFDIR|0755, stx_size=4096, ...}) = 0 statx(AT_FDCWD, "/", AT_STATX_SYNC_AS_STAT|AT_NO_AUTOMOUNT, STATX_BASIC_STATS, {stx_mask=STATX_BASIC_STATS|STATX_MNT_ID, stx_attributes=0, stx_mode=S_IFDIR|0755, stx_size=4096, ...}) = 0 write(2, "D: ", 3) = 3 write(2, "0x0000fd00 4096 0"..., 62) = 62 The statfs() is on a strange path, and then I dunno, some emulation going on which gets it wrong, mayhap?
So. When the same thing works on bare metal but fails when emulated, I'm inclined to blame the emulator. Short summary: it seems that statvfs() in this situation (inside chroot and all) returns with ST_RDONLY set in f_flag or 0 in f_bavail, either would seem incorrect.
Panu, I don't suppose it would be possible to distil this down to a minimal reproducer? My best attempt was below but I wasn't able to reproduce the problem with qemu-user-5.0.0-5.fc33.x86_64. What version of qemu-user is supposed to cause the problem? --- #include <stdio.h> #include <stdlib.h> #include <sys/stat.h> #include <sys/vfs.h> int main () { struct statfs buffs; struct stat buf; if (statfs ("/", &buffs) == -1) { perror ("statfs"); exit (1); } printf ("f_bavail: %d\n", buffs.f_bavail); if (stat ("/", &buf) == -1) { perror ("stat"); exit (1); } printf ("st_size: %d\n", buf.st_size); exit (0); }
qemu-user-static-5.1.0-5.fc33.x86_64 is the version I used to reproduce. The issue occurs inside a chroot (in fact a nested one), which is likely relevant for reproducing. I didn't try this, but: # dnf -y --installroot /var/tmp/main-root --releasever 34 --forcearch armv7hl install dnf --setopt=tsflags=nocontexts --disablerepo='*' --enablerepo fedora --enablerepo updates # chroot /var/tmp/main-root/ # run your test program If that doesn't reproduce it, add the nested chroot to the mix, ie in the test-program before stat'ing: mkdir('/sub-root') chroot('/sub-root')
The nested chroot is very likely the key to this. In the initial fedora-devel thread, it's mentioned that disabling bootstrap mode works around the issue: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/NNP6TZNG2ZRQ3XCKCJWFTL3XIABOBZJW/#NNP6TZNG2ZRQ3XCKCJWFTL3XIABOBZJW
> In the initial fedora-devel thread, it's mentioned that disabling bootstrap > mode works around the issue: When you disable bootstrap, the build chroot is installed by the system-default DNF/RPM stack, so the emulation isn't in action at all. That said, I'm not sure the second (nested) chroot is needed. It should be enough to just use one chroot to actually run statfs() through the emulation layer.
Ah, it of course depends on what RPM is doing ... if it calls chroot() internally with `--root DIRECTORY`, another chroot level might really be needed.
Yes, rpm does it's own chroot() when --root / --installroot is used.
This bug appears to have been reported against 'rawhide' during the Fedora 34 development cycle. Changing version to 34.
FTR, after moving to F34 I fail to reproduce the original issue.
Ok, as I'm the submitter of this bug - I think it is fair to close this now (we moved Copr builders to F34). But feel free to reopen in case you want to fix F33.