Hide Forgot
Description of problem: I initially was going to file this against the kernel, but I'm leaning more now towards supermin doing something funky. After upgrading the kernel from 5.3.7 to 5.3.8, we've noticed a regression where our supermin appliance only sees one file in `/etc/pki/rpm-gpg`, instead of all the GPG keys. Version-Release number of selected component (if applicable): Reproduces with supermin-5.1.20-8.fc31.x86_64 + kernel-5.3.[89]. Does not reproduce with supermin-5.1.20-8.fc31.x86_64 + kernel-5.3.7. How reproducible: 100% Steps to Reproduce: ``` $ export SUPERMIN_KERNEL=/usr/lib/modules/5.3.8-300.fc31.x86_64/vmlinuz $ export SUPERMIN_MODULES=/usr/lib/modules/5.3.8-300.fc31.x86_64 $ rpm -q fedora-gpg-keys fedora-gpg-keys-31-1.noarch $ ls -la /etc/pki/rpm-gpg | wc -l 218 $ supermin --prepare --use-installed -o supermin.prepare bash coreutils fedora-gpg-keys util-linux $ tar zcf supermin.prepare/init.tar.gz ./init $ supermin --build supermin.prepare --size 5G -f ext2 -o supermin.build $ qemu-kvm -nodefaults -nographic -kernel supermin.build/kernel -initrd supermin.build/initrd -hda supermin.build/root -serial stdio -append "console=ttyS0 root=/dev/sda" ... bash-5.0# ls -la /etc/pki/rpm-gpg ls: reading directory '/etc/pki/rpm-gpg': Input/output error total 24 drwxr-xr-x 2 root root 20480 Nov 7 22:15 . drwxrwxr-x 6 jlebon jlebon 4096 Nov 8 16:07 .. lrwxrwxrwx 1 root root 29 Nov 8 16:07 RPM-GPG-KEY-31-fedora -> RPM-GPG-KEY-fedora-31-primary bash-5.0# ``` Actual results: Only one broken symlink. Expected results: All the GPG keys + symlinks. Additional info: What's interesting is that running with 5.3.7, all the files show up, but it seems like symlinks show up twice: once with the absolute path, and once without: ``` bash-5.0# cd /etc/pki/rpm-gpg bash-5.0# ls -la total 204 drwxr-xr-x 2 root root 20480 Nov 7 22:15 . drwxrwxr-x 6 jlebon jlebon 4096 Nov 8 16:14 .. lrwxrwxrwx 1 root root 29 Nov 8 16:14 /etc/pki/rpm-gpg/RPM-GPG-KEY-31-fedora -> RPM-GPG-KEY-fedora-31-primary lrwxrwxrwx 1 root root 29 Nov 8 16:14 /etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-10-i386 -> RPM-GPG-KEY-fedora-10-primary lrwxrwxrwx 1 root root 29 Nov 8 16:14 /etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-10-ppc -> RPM-GPG-KEY-fedora-10-primary lrwxrwxrwx 1 root root 29 Nov 8 16:14 /etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-10-ppc64 -> RPM-GPG-KEY-fedora-10-primary lrwxrwxrwx 1 root root 29 Nov 8 16:14 /etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-10-x86_64 -> RPM-GPG-KEY-fedora-10-primary ... lrwxrwxrwx 1 root root 29 Nov 8 16:14 RPM-GPG-KEY-31-fedora -> RPM-GPG-KEY-fedora-31-primary lrwxrwxrwx 1 root root 29 Nov 8 16:14 RPM-GPG-KEY-fedora-10-i386 -> RPM-GPG-KEY-fedora-10-primary lrwxrwxrwx 1 root root 29 Nov 8 16:14 RPM-GPG-KEY-fedora-10-ppc -> RPM-GPG-KEY-fedora-10-primary lrwxrwxrwx 1 root root 29 Nov 8 16:14 RPM-GPG-KEY-fedora-10-ppc64 -> RPM-GPG-KEY-fedora-10-primary -rw-r--r-- 1 root root 2380 Oct 11 18:50 RPM-GPG-KEY-fedora-10-primary lrwxrwxrwx 1 root root 29 Nov 8 16:14 RPM-GPG-KEY-fedora-10-x86_64 -> RPM-GPG-KEY-fedora-10-primary ... bash-5.0# ls -la /etc/pki/rpm-gpg | wc -l 384 ``` Running e2fsck on `root` complains about this too: ``` Entry '/etc/pki/rpm-gpg/RPM-GPG-KEY-31-fedora' in /etc/pki/rpm-gpg (355) has illegal characters in its name. Fix<y>? no Entry '/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-10-i386' in /etc/pki/rpm-gpg (355) has illegal characters in its name. Fix<y>? no Entry '/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-10-ppc' in /etc/pki/rpm-gpg (355) has illegal characters in its name. Fix<y>? no Entry '/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-10-ppc64' in /etc/pki/rpm-gpg (355) has illegal characters in its name. Fix<y>? no Entry '/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-10-x86_64' in /etc/pki/rpm-gpg (355) has illegal characters in its name. Fix<y>? no ... ``` So I think what's happening there is that before 5.3.8, the kernel tolerated these errors, but now it doesn't.
We've actually had issues with symlinks before, see bug 1470157 and https://www.redhat.com/archives/libguestfs/2017-July/msg00078.html However as a result of that patch we are now using the ext2fs_symlink library function from e2fsprogs, which I hope always creates symlinks correctly (unless e2fsprogs has a bug). Eric - any ideas?
While ext4 has a library to make filesystems from userspace (unlike many other filesystems), it's clearly a 2nd tier thing. Is there a reason for supermin to do this at all versus say just booting from a 9p (or https://virtio-fs.gitlab.io/) rootfs?
9p isn't in RHEL and virtio-fs didn't exist. Note however that mke2fs itself uses the same library to build filesystems (using the mke2fs -d option) so it's not a second tier method.
> 9p isn't in RHEL and virtio-fs didn't exist. Yeah =( I am really, really looking forward to virtio-fs landing. > Note however that mke2fs itself uses the same library to build filesystems (using the mke2fs -d option) There's a difference between making a filesystem "structure" (superblocks etc.) versus actually writing *content*. It's the latter that I think is "2nd tier" - bigger picture, most people use the Linux kernel itself to write filesystem content; this is obviously what libguestfs is designed to do, we're just using this ext2 image to bootstrap. I think OpenEmbedded also uses the ext4 lib, but personally I'm way happier with the flexibility of just requiring KVM.
cc: Lukas too but TBH I'm lacking 99% of the context I probably need to have a clue here, as I have no idea what supermin does or how it does it...
(In reply to Eric Sandeen from comment #5) > cc: Lukas too but TBH I'm lacking 99% of the context I probably need to have > a clue here, as I have no idea what supermin does or how it does it... It's building a small ext4 filesystem in a file to use as a bootable appliance. It uses e2fsprogs for this. There's some issue about whether symlinks are being created correctly, since kernel 5.3.8 apparently cannot read them (< 5.3.8 apparently could read them fine). A simple example which doesn't need root is: http://libguestfs.org/supermin.1.html#EXAMPLE Note I've not investigated this bug at all so far because I've just got back from 2 weeks of conferences. (In reply to Colin Walters from comment #4) > There's a difference between making a filesystem "structure" (superblocks > etc.) versus actually writing *content*. mke2fs -d really builds a full filesystem with content.
Just hit this same issue on Arch Linux. supermin ext2 images have always seemed a bit weird to me. For example, this is on an older LTS kernel: $ cd /var/tmp $ wget http://download.libguestfs.org/binaries/appliance/appliance-1.40.1.tar.xz $ tar -xf appliance-1.40.1.tar.xz $ sudo mount -o loop appliance/root /mnt/tmp $ ls -l /mnt/tmp total 84 lrwxrwxrwx 1 root root 7 Nov 14 03:23 /bin -> usr/bin lrwxrwxrwx 1 root root 7 Mar 4 2019 bin -> usr/bin dr-xr-xr-x 2 root root 4096 Mar 4 2019 boot drwxr-xr-x 2 root root 4096 Mar 4 2019 dev drwxr-xr-x 58 gws gws 4096 Jan 18 2019 etc drwxr-xr-x 2 root root 4096 Dec 13 2018 home -rwxr-xr-x 1 gws gws 7120 Jan 17 2019 init lrwxrwxrwx 1 root root 7 Nov 14 03:23 /lib -> usr/lib lrwxrwxrwx 1 root root 7 Mar 4 2019 lib -> usr/lib lrwxrwxrwx 1 root root 7 Nov 14 03:23 /lib64 -> usr/lib lrwxrwxrwx 1 root root 9 Mar 4 2019 lib64 -> usr/lib64 drwx------ 2 root root 16384 Mar 4 2019 lost+found drwxr-xr-x 2 root root 4096 Jul 13 2018 media drwxr-xr-x 2 root root 4096 Dec 14 2018 mnt drwxr-xr-x 2 root root 4096 Jul 13 2018 opt dr-xr-xr-x 2 root root 4096 Feb 4 2019 proc dr-xr-x--- 2 root root 4096 Jan 12 2019 root drwxr-xr-x 12 root root 4096 Mar 4 2019 run lrwxrwxrwx 1 root root 7 Nov 14 03:23 /sbin -> usr/bin lrwxrwxrwx 1 root root 8 Mar 4 2019 sbin -> usr/sbin drwxr-xr-x 2 root root 4096 Jul 13 2018 srv dr-xr-xr-x 2 root root 4096 Feb 4 2019 sys drwxrwxrwt 2 root root 4096 Mar 4 2019 tmp drwxr-xr-x 12 gws gws 4096 Jan 18 2019 usr drwxr-xr-x 19 root root 4096 Feb 1 2019 var Notice those '/' characters (forward slashes) in the directory entries? They just look wrong. I'd guess they are the "illegal characters" and the source of the problem.
I haven't bisected it to confirm, but this commit appeared in 5.3.8 and looks relevant: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/fs/readdir.c?id=8a23eb804ca4f2be909e372cf5a9e7b30ae476cd
Forgotten about this bug, thanks for reminding me. Yes I'm able to reproduce it easily with kernel 5.4.0. Here's a small reproducer: $ supermin --prepare bash util-linux -o /tmp/supermin.d $ supermin --build /tmp/supermin.d -f ext2 -o /tmp/appliance.d $ ll /tmp/appliance.d/root -rw-r--r--. 1 rjones rjones 4294967296 Nov 26 08:38 /tmp/appliance.d/root $ guestfish --ro -a /tmp/appliance.d/root -m /dev/sda ll / libguestfs: error: ll: ls: reading directory '/sysroot/': Input/output error Copying the /tmp/appliance.d/root file to a < 5.3.8 kernel machine shows: lrwxrwxrwx 1 root root 7 Nov 26 08:43 /bin -> usr/bin lrwxrwxrwx 1 root root 7 Nov 26 08:43 /lib -> usr/lib lrwxrwxrwx 1 root root 9 Nov 26 08:43 /lib64 -> usr/lib64 lrwxrwxrwx 1 root root 8 Nov 26 08:43 /sbin -> usr/sbin lrwxrwxrwx 1 root root 7 Nov 26 08:38 bin -> usr/bin which are clearly invalid directory entries. It does indeed look as if it's a bug in supermin. Also the patch in comment 8 does look very relevant.
Created attachment 1639725 [details] 0001-ext2-Build-symbolic-links-correctly-RHBZ-1770304.patch This is only lightly tested but works for me so far. I will do a bit more testing this morning and post it formally on the mailing list if it works.
Patch posted: https://www.redhat.com/archives/libguestfs/2019-November/msg00262.html
Patch works for me. Thanks!
Fedora 29 changed to end-of-life (EOL) status on 2019-11-26. Fedora 29 is no longer maintained, which means that it will not receive any further security or bug fix updates. As a result we are closing this bug. If you can reproduce this bug against a currently maintained version of Fedora please feel free to reopen this bug against that version. If you are unable to reopen this bug, please file a new report against the current release. If you experience problems, please add a comment to this bug. Thank you for reporting this bug and we are sorry it could not be fixed.
This bug is opened against f31, not f29.
Upstream in https://github.com/libguestfs/supermin/commit/487e8c964078f823646d3b0584745ab7d0ef99ca
FEDORA-2019-f4bf176b5c has been submitted as an update to Fedora 30. https://bodhi.fedoraproject.org/updates/FEDORA-2019-f4bf176b5c
FEDORA-2019-e8415f65c9 has been submitted as an update to Fedora 31. https://bodhi.fedoraproject.org/updates/FEDORA-2019-e8415f65c9
supermin-5.1.20-10.fc30 has been pushed to the Fedora 30 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-f4bf176b5c
supermin-5.1.20-10.fc31 has been pushed to the Fedora 31 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-e8415f65c9
See also https://bugzilla.redhat.com/show_bug.cgi?id=1781803
*** Bug 1781803 has been marked as a duplicate of this bug. ***
FEDORA-2019-3454e38e8c has been submitted as an update to Fedora 30. https://bodhi.fedoraproject.org/updates/FEDORA-2019-3454e38e8c
FEDORA-2019-0fb47d9231 has been submitted as an update to Fedora 31. https://bodhi.fedoraproject.org/updates/FEDORA-2019-0fb47d9231
supermin-5.1.20-11.fc30 has been pushed to the Fedora 30 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-3454e38e8c
supermin-5.1.20-11.fc31 has been pushed to the Fedora 31 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-0fb47d9231
supermin-5.1.20-11.fc30 has been pushed to the Fedora 30 stable repository. If problems still persist, please make note of it in this bug report.
supermin-5.1.20-11.fc31 has been pushed to the Fedora 31 stable repository. If problems still persist, please make note of it in this bug report.