Bug 1895363 - Dnf transaction calculated in --forcearch mode thinks there's no space in / partition
Summary: Dnf transaction calculated in --forcearch mode thinks there's no space in / p...
Keywords:
Status: CLOSED WORKSFORME
Alias: None
Product: Fedora
Classification: Fedora
Component: qemu
Version: 34
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Fedora Virtualization Maintainers
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-11-06 13:32 UTC by Pavel Raiskup
Modified: 2021-06-15 06:36 UTC (History)
28 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-06-15 06:36:05 UTC
Type: Bug


Attachments (Terms of Use)

Description Pavel Raiskup 2020-11-06 13:32:17 UTC
This can be easily reproduced on Fedora 33 x86_64 using mock:
$ mock -r fedora-rawhide-armhfp --shell
....

Observed by @jkadlcik, while running the Copr testsuite.

Steps to reproduce without mock /on F33 x86_64/

#> # quite some free space ....
#> df -h /var/tmp/
Filesystem             Size  Used Avail Use% Mounted on
/dev/mapper/luks-....  477G  374G  103G  79% /
#> # install bootstrap chroot
#> dnf -y --installroot /var/tmp/main-root --releasever 34 --forcearch armv7hl install dnf --setopt=tsflags=nocontexts --disablerepo='*' --enablerepo fedora --enablerepo updates

#> # Necessary changes to make dnf in chroot work 
# rm -f /var/tmp/main-root-rawhide/etc/resolv.conf
# cp /etc/resolv.conf /var/tmp/main-root-rawhide/etc/resolv.conf
# mount --bind /dev/urandom /var/tmp/main-root-rawhide/dev/urandom

# chroot /var/tmp/main-root/
bash-5.0# /usr/bin/dnf --installroot /sub-root --releasever 34 --forcearch armv7hl install filesystem --setopt=tsflags=nocontexts --disablerepo='*' --enablerepo fedora --enablerepo updates         
Failed to set locale, defaulting to C.UTF-8
Fedora 34 - armhfp - Updates                                                                                                                                                                                2.3 MB/s |  69 MB     00:30    
Fedora 34 - armhfp                                                                                                                                                                                          324 kB/s | 1.6 MB     00:05    
Last metadata expiration check: 0:01:34 ago on Fri Nov  6 13:27:44 2020.
Dependencies resolved.
============================================================================================================================================================================================================================================
 Package                                                               Architecture                                    Version                                                        Repository                                       Size
============================================================================================================================================================================================================================================
Installing:
 filesystem                                                            armv7hl                                         3.14-4.fc34                                                    updates                                         1.1 M
Installing dependencies:
 basesystem                                                            noarch                                          11-10.fc33                                                     updates                                         6.8 k
 bash                                                                  armv7hl                                         5.0.17-2.fc33                                                  updates                                         1.6 M
 fedora-gpg-keys                                                       noarch                                          34-0.8                                                         updates                                         107 k
 fedora-release                                                        noarch                                          34-0.9                                                         updates                                          12 k
 fedora-release-common                                                 noarch                                          34-0.9                                                         updates                                          21 k
 fedora-release-identity-basic                                         noarch                                          34-0.9                                                         updates                                          13 k
 fedora-repos                                                          noarch                                          34-0.8                                                         updates                                          11 k
 fedora-repos-rawhide                                                  noarch                                          34-0.8                                                         updates                                          10 k
 glibc                                                                 armv7hl                                         2.32.9000-12.fc34                                              updates                                         3.1 M
 glibc-common                                                          armv7hl                                         2.32.9000-12.fc34                                              updates                                         1.8 M
 glibc-minimal-langpack                                                armv7hl                                         2.32.9000-12.fc34                                              updates                                          93 k
 libgcc                                                                armv7hl                                         10.2.1-6.fc34                                                  updates                                         102 k
 ncurses-base                                                          noarch                                          6.2-3.20200222.fc33                                            updates                                          60 k
 ncurses-libs                                                          armv7hl                                         6.2-3.20200222.fc33                                            updates                                         285 k
 setup                                                                 noarch                                          2.13.7-2.fc33                                                  updates                                         142 k
 tzdata                                                                noarch                                          2020d-1.fc34                                                   updates                                         430 k

Transaction Summary
============================================================================================================================================================================================================================================
Install  17 Packages

Total download size: 8.9 M
Installed size: 31 M
Is this ok [y/N]: y
Downloading Packages:
(1/17): basesystem-11-10.fc33.noarch.rpm                                                                                                                                                                     34 kB/s | 6.8 kB     00:00    
(2/17): fedora-release-34-0.9.noarch.rpm                                                                                                                                                                    315 kB/s |  12 kB     00:00    
(3/17): fedora-release-common-34-0.9.noarch.rpm                                                                                                                                                             492 kB/s |  21 kB     00:00    
(4/17): fedora-gpg-keys-34-0.8.noarch.rpm                                                                                                                                                                   363 kB/s | 107 kB     00:00    
(5/17): fedora-release-identity-basic-34-0.9.noarch.rpm                                                                                                                                                     468 kB/s |  13 kB     00:00    
(6/17): fedora-repos-34-0.8.noarch.rpm                                                                                                                                                                      400 kB/s |  11 kB     00:00    
(7/17): fedora-repos-rawhide-34-0.8.noarch.rpm                                                                                                                                                              342 kB/s |  10 kB     00:00    
(8/17): filesystem-3.14-4.fc34.armv7hl.rpm                                                                                                                                                                  1.2 MB/s | 1.1 MB     00:00    
(9/17): bash-5.0.17-2.fc33.armv7hl.rpm                                                                                                                                                                      1.3 MB/s | 1.6 MB     00:01    
(10/17): glibc-minimal-langpack-2.32.9000-12.fc34.armv7hl.rpm                                                                                                                                               1.1 MB/s |  93 kB     00:00    
(11/17): libgcc-10.2.1-6.fc34.armv7hl.rpm                                                                                                                                                                   1.1 MB/s | 102 kB     00:00    
(12/17): ncurses-base-6.2-3.20200222.fc33.noarch.rpm                                                                                                                                                        421 kB/s |  60 kB     00:00    
(13/17): ncurses-libs-6.2-3.20200222.fc33.armv7hl.rpm                                                                                                                                                       1.1 MB/s | 285 kB     00:00    
(14/17): setup-2.13.7-2.fc33.noarch.rpm                                                                                                                                                                     1.0 MB/s | 142 kB     00:00    
(15/17): tzdata-2020d-1.fc34.noarch.rpm                                                                                                                                                                     1.1 MB/s | 430 kB     00:00    
(16/17): glibc-2.32.9000-12.fc34.armv7hl.rpm                                                                                                                                                                1.3 MB/s | 3.1 MB     00:02    
(17/17): glibc-common-2.32.9000-12.fc34.armv7hl.rpm                                                                                                                                                         1.2 MB/s | 1.8 MB     00:01    
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                                                                       2.3 MB/s | 8.9 MB     00:03     
warning: [fd 19]: Header V4 RSA/SHA256 Signature, key ID 45719a39: NOKEY
Fedora 34 - armhfp - Updates                                                                                                                                                                                660 kB/s | 1.6 kB     00:00    
Importing GPG key 0x45719A39:
 Userid     : "Fedora (34) <fedora-34-primary@fedoraproject.org>"
 Fingerprint: 8C5B A699 0BDB 26E1 9F2A 1A80 1161 AE69 4571 9A39
 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-34-armhfp
Is this ok [y/N]: y
Key imported successfully
Running transaction check
Transaction check succeeded.
Running transaction test
The downloaded packages were saved in cache until the next successful transaction.
You can remove cached packages by executing 'dnf clean packages'.
Error: Transaction test error:
  installing package fedora-release-identity-basic-34-0.9.noarch needs 44KB more space on the / filesystem
  installing package tzdata-2020d-1.fc34.noarch needs 7MB more space on the / filesystem
  installing package ncurses-base-6.2-3.20200222.fc33.noarch needs 7MB more space on the / filesystem
  installing package fedora-gpg-keys-34-0.8.noarch needs 8MB more space on the / filesystem
  installing package libgcc-10.2.1-6.fc34.armv7hl needs 9MB more space on the / filesystem
  installing package fedora-release-34-0.9.noarch needs 9MB more space on the / filesystem
  installing package fedora-release-common-34-0.9.noarch needs 9MB more space on the / filesystem
  installing package fedora-repos-rawhide-34-0.8.noarch needs 9MB more space on the / filesystem
  installing package fedora-repos-34-0.8.noarch needs 9MB more space on the / filesystem
  installing package setup-2.13.7-2.fc33.noarch needs 10MB more space on the / filesystem
  installing package filesystem-3.14-4.fc34.armv7hl needs 14MB more space on the / filesystem
  installing package basesystem-11-10.fc33.noarch needs 14MB more space on the / filesystem
  installing package glibc-minimal-langpack-2.32.9000-12.fc34.armv7hl needs 15MB more space on the / filesystem
  installing package glibc-common-2.32.9000-12.fc34.armv7hl needs 23MB more space on the / filesystem
  installing package glibc-2.32.9000-12.fc34.armv7hl needs 39MB more space on the / filesystem
  installing package ncurses-libs-6.2-3.20200222.fc33.armv7hl needs 40MB more space on the / filesystem
  installing package bash-5.0.17-2.fc33.armv7hl needs 48MB more space on the / filesystem

Error Summary
-------------
Disk Requirements:
   At least 48MB more space needed on the / filesystem.

bash-5.0#

Comment 1 Panu Matilainen 2020-11-20 11:19:33 UTC
FWIW, I'm not able to reproduce with the above instructions, in mock or otherwise (after correcting /var/tmp/main-root-rawhide/ vs /var/tmp/main-root/ inconsistencies in the instructions). So there seems to be some missing ingredient.

But with the non-mock instructions I get this, which might be related...
bash-5.0# df
/usr/bin/df: cannot read table of mounted file systems: No such file or directory

> This can be easily reproduced on Fedora 33 x86_64 using mock:
> $ mock -r fedora-rawhide-armhfp --shell
> ...

Is that "mock -r fedora-rawhide-armhfp --shell" alone supposed to reproduce it, or does "..." mean some "business as usual" commands that might be relevant after all?

Comment 2 Pavel Raiskup 2020-11-20 11:46:21 UTC
> after correcting /var/tmp/main-root-rawhide/ vs /var/tmp/main-root/

Yes, sorry.  I tested this again onw on my F33, so I corrected the
instructions:

    rpm -q rpm dnf
    rpm-4.16.0-1.fc33.x86_64
    dnf-4.4.0-3.fc33.noarch

Steps to reproduce:

sudo su -
dnf -y --installroot /var/tmp/main-root --releasever 34 --forcearch armv7hl install dnf --setopt=tsflags=nocontexts --disablerepo='*' --enablerepo fedora --enablerepo updates
# the resolv.conf symlink is not created anymore by systemd, so no need to
# remove it now
cp /etc/resolv.conf /var/tmp/main-root/etc/resolv.conf
touch /var/tmp/main-root/dev/urandom
mount --bind /dev/urandom /var/tmp/main-root/dev/urandom
chroot /var/tmp/main-root/
bash-5.0# /usr/bin/dnf -y --installroot /sub-root --releasever 34 --forcearch armv7hl install filesystem --setopt=tsflags=nocontexts --disablerepo='*' --enablerepo fedora --enablerepo updates
...
Error: Transaction test error:
  installing package fedora-release-identity-basic-34-0.9.noarch needs 44KB more space on the / filesystem
...
Error Summary
-------------
Disk Requirements:
   At least 48MB more space needed on the / filesystem.


> Is that "mock -r fedora-rawhide-armhfp --shell" alone supposed to reproduce it

Yes, but I guess I should suggest 'mock -r fedora-rawhide-armhfp --scrub=all'
first, and turn on (if explicitly disabled) the bootstrap.  The mock process
fails when it tries to install the build chroot from the (emulated) bootstrap
chroot.

Comment 3 Panu Matilainen 2020-11-20 13:08:26 UTC
This seems rather unstable at best. I can now fairly reliably reproduce the first failure (outside mock), but what happens after than varies a lot. Sometimes the second attempt at the last command succeeds, sometimes it doesn't, and so on.

When it fails, rpm on debug logging shows:
D: computing file dispositions
D: 0x0000fd00     4096            0      4062891 rotational:-1 /

Which says 4096 blocksize, 0 available blocks and 4062891 available inodes on the / filesystem. Zero available blocks could be from read-only filesystem indicated by statvfs(), but other than that it's just whatever statvfs() returns, and rpm is right to stop when there's no space indicated (or the media is read-only, which is mapped to 0 avail by rpm)

What does this forcearch-thing really do? That's where I would look at first.

Comment 4 Panu Matilainen 2020-11-20 13:41:42 UTC
Figured I can strace the thing from the outside. This is what that activity looks like when run from the host (with rpm -Uvv --ignorearch):

stat("/var/lib/rpm", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
statfs("/", {f_type=EXT2_SUPER_MAGIC, f_bsize=4096, f_blocks=17931038, f_bfree=4107696, f_bavail=3186096, f_files=4587520, f_ffree=4078319, f_fsid={val=[3681299638, 2236116973]}, f_namelen=255, f_frsize=4096, f_flags=ST_VALID|ST_RELATIME}) = 0
stat("/", {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0
stat("/", {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0
write(2, "D: ", 3)                      = 3
write(2, "0x0000fd00     4096      3186096"..., 62) = 62


What happens in the qemu-static process is something quite different:

statx(AT_FDCWD, "/var/lib/rpm", AT_STATX_SYNC_AS_STAT|AT_NO_AUTOMOUNT, STATX_BASIC_STATS, {stx_mask=STATX_BASIC_STATS|STATX_MNT_ID, stx_attributes=0, stx_mode=S_IFDIR|0755, stx_size=4096, ...}) = 0
mmap(NULL, 1052672, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f1de2bc6000
futex(0x7e7dc8, FUTEX_WAKE, 2147483647) = 1
access("/usr/qemu-arm/usr/lib/", F_OK)  = -1 ENOENT (No such file or directory)
statfs("/usr/lib/", {f_type=EXT2_SUPER_MAGIC, f_bsize=4096, f_blocks=17931038, f_bfree=3938041, f_bavail=3016441, f_files=4587520, f_ffree=4062853, f_fsid={val=[3681299638, 2236116973]}, f_namelen=255, f_frsize=4096, f_flags=ST_VALID|ST_RELATIME}) = 0
statx(AT_FDCWD, "/usr/lib/", AT_STATX_SYNC_AS_STAT|AT_NO_AUTOMOUNT, STATX_BASIC_STATS, {stx_mask=STATX_BASIC_STATS|STATX_MNT_ID, stx_attributes=0, stx_mode=S_IFDIR|0755, stx_size=4096, ...}) = 0
statx(AT_FDCWD, "/usr", AT_STATX_SYNC_AS_STAT|AT_SYMLINK_NOFOLLOW|AT_NO_AUTOMOUNT, STATX_BASIC_STATS, {stx_mask=STATX_BASIC_STATS|STATX_MNT_ID, stx_attributes=0, stx_mode=S_IFDIR|0755, stx_size=4096, ...}) = 0
statx(AT_FDCWD, "/usr/lib", AT_STATX_SYNC_AS_STAT|AT_SYMLINK_NOFOLLOW|AT_NO_AUTOMOUNT, STATX_BASIC_STATS, {stx_mask=STATX_BASIC_STATS|STATX_MNT_ID, stx_attributes=0, stx_mode=S_IFDIR|0755, stx_size=4096, ...}) = 0
statx(AT_FDCWD, "/usr", AT_STATX_SYNC_AS_STAT|AT_NO_AUTOMOUNT, STATX_BASIC_STATS, {stx_mask=STATX_BASIC_STATS|STATX_MNT_ID, stx_attributes=0, stx_mode=S_IFDIR|0755, stx_size=4096, ...}) = 0
statx(AT_FDCWD, "/", AT_STATX_SYNC_AS_STAT|AT_NO_AUTOMOUNT, STATX_BASIC_STATS, {stx_mask=STATX_BASIC_STATS|STATX_MNT_ID, stx_attributes=0, stx_mode=S_IFDIR|0755, stx_size=4096, ...}) = 0
write(2, "D: ", 3)                      = 3
write(2, "0x0000fd00     4096            0"..., 62) = 62


The statfs() is on a strange path, and then I dunno, some emulation going on which gets it wrong, mayhap?

Comment 5 Panu Matilainen 2020-11-20 14:01:43 UTC
So. When the same thing works on bare metal but fails when emulated, I'm inclined to blame the emulator. 
Short summary: it seems that statvfs() in this situation (inside chroot and all) returns with ST_RDONLY set in f_flag or 0 in f_bavail, either would seem incorrect.

Comment 6 Richard W.M. Jones 2020-11-21 08:47:50 UTC
Panu, I don't suppose it would be possible to distil this down to a minimal reproducer?

My best attempt was below but I wasn't able to reproduce the problem with
qemu-user-5.0.0-5.fc33.x86_64.  What version of qemu-user is supposed to
cause the problem?

---
#include <stdio.h>
#include <stdlib.h>
#include <sys/stat.h>
#include <sys/vfs.h>

int
main ()
{
  struct statfs buffs;
  struct stat buf;
  if (statfs ("/", &buffs) == -1) { perror ("statfs"); exit (1); }
  printf ("f_bavail: %d\n", buffs.f_bavail);
  if (stat ("/", &buf) == -1) { perror ("stat"); exit (1); }
  printf ("st_size: %d\n", buf.st_size);
  exit (0);
}

Comment 7 Panu Matilainen 2020-11-23 07:02:18 UTC
qemu-user-static-5.1.0-5.fc33.x86_64 is the version I used to reproduce.

The issue occurs inside a chroot (in fact a nested one), which is likely relevant for reproducing. I didn't try this, but:

# dnf -y --installroot /var/tmp/main-root --releasever 34 --forcearch armv7hl install dnf --setopt=tsflags=nocontexts --disablerepo='*' --enablerepo fedora --enablerepo updates
# chroot /var/tmp/main-root/
# run your test program

If that doesn't reproduce it, add the nested chroot to the mix, ie in the test-program before stat'ing:
  mkdir('/sub-root')
  chroot('/sub-root')

Comment 8 Panu Matilainen 2020-11-23 08:02:16 UTC
The nested chroot is very likely the key to this.

In the initial fedora-devel thread, it's mentioned that disabling bootstrap mode works around the issue:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/NNP6TZNG2ZRQ3XCKCJWFTL3XIABOBZJW/#NNP6TZNG2ZRQ3XCKCJWFTL3XIABOBZJW

Comment 9 Pavel Raiskup 2020-11-23 08:46:15 UTC
> In the initial fedora-devel thread, it's mentioned that disabling bootstrap
> mode works around the issue:

When you disable bootstrap, the build chroot is installed by the
system-default DNF/RPM stack, so the emulation isn't in action at all.

That said, I'm not sure the second (nested) chroot is needed.  It should
be enough to just use one chroot to actually run statfs() through the
emulation layer.

Comment 10 Pavel Raiskup 2020-11-23 09:11:14 UTC
Ah, it of course depends on what RPM is doing ... if it calls chroot()
internally with `--root DIRECTORY`, another chroot level might really
be needed.

Comment 11 Panu Matilainen 2020-11-23 10:08:02 UTC
Yes, rpm does it's own chroot() when --root / --installroot is used.

Comment 12 Ben Cotton 2021-02-09 15:23:07 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 34 development cycle.
Changing version to 34.

Comment 13 Pavel Raiskup 2021-06-15 06:33:33 UTC
FTR, after moving to F34 I fail to reproduce the original issue.

Comment 14 Pavel Raiskup 2021-06-15 06:36:05 UTC
Ok, as I'm the submitter of this bug - I think it is fair to close this now
(we moved Copr builders to F34).  But feel free to reopen in case you want
to fix F33.


Note You need to log in before you can comment on or make changes to this bug.