Bug 1770304 - ext2 filesystem supermin assembles doesn't write symlinks correctly
Summary: ext2 filesystem supermin assembles doesn't write symlinks correctly
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: supermin
Version: 31
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Richard W.M. Jones
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
: 1781803 (view as bug list)
Depends On:
Blocks: 1776719
TreeView+ depends on / blocked
 
Reported: 2019-11-08 16:48 UTC by Jonathan Lebon
Modified: 2020-01-05 00:39 UTC (History)
8 users (show)

Fixed In Version: supermin-5.1.20-11.fc30 supermin-5.1.20-11.fc31
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1776719 (view as bug list)
Environment:
Last Closed: 2020-01-03 20:35:34 UTC
Type: Bug


Attachments (Terms of Use)
0001-ext2-Build-symbolic-links-correctly-RHBZ-1770304.patch (1.39 KB, patch)
2019-11-26 09:02 UTC, Richard W.M. Jones
no flags Details | Diff

Description Jonathan Lebon 2019-11-08 16:48:35 UTC
Description of problem:

I initially was going to file this against the kernel, but I'm leaning more now towards supermin doing something funky. After upgrading the kernel from 5.3.7 to 5.3.8, we've noticed a regression where our supermin appliance only sees one file in `/etc/pki/rpm-gpg`, instead of all the GPG keys.

Version-Release number of selected component (if applicable):

Reproduces with supermin-5.1.20-8.fc31.x86_64 + kernel-5.3.[89].
Does not reproduce with supermin-5.1.20-8.fc31.x86_64 + kernel-5.3.7.

How reproducible:

100%

Steps to Reproduce:

```
$ export SUPERMIN_KERNEL=/usr/lib/modules/5.3.8-300.fc31.x86_64/vmlinuz
$ export SUPERMIN_MODULES=/usr/lib/modules/5.3.8-300.fc31.x86_64
$ rpm -q fedora-gpg-keys
fedora-gpg-keys-31-1.noarch
$ ls -la /etc/pki/rpm-gpg | wc -l
218
$ supermin --prepare --use-installed -o supermin.prepare bash coreutils fedora-gpg-keys util-linux
$ tar zcf supermin.prepare/init.tar.gz ./init
$ supermin --build supermin.prepare --size 5G -f ext2 -o supermin.build
$ qemu-kvm -nodefaults -nographic -kernel supermin.build/kernel -initrd supermin.build/initrd -hda supermin.build/root -serial stdio -append "console=ttyS0 root=/dev/sda"
...
bash-5.0# ls -la /etc/pki/rpm-gpg
ls: reading directory '/etc/pki/rpm-gpg': Input/output error
total 24
drwxr-xr-x 2 root   root   20480 Nov  7 22:15 .
drwxrwxr-x 6 jlebon jlebon  4096 Nov  8 16:07 ..
lrwxrwxrwx 1 root   root      29 Nov  8 16:07 RPM-GPG-KEY-31-fedora -> RPM-GPG-KEY-fedora-31-primary
bash-5.0#
```

Actual results:

Only one broken symlink.

Expected results:

All the GPG keys + symlinks.

Additional info:

What's interesting is that running with 5.3.7, all the files show up, but it seems like symlinks show up twice: once with the absolute path, and once without:

```
bash-5.0# cd /etc/pki/rpm-gpg
bash-5.0# ls -la
total 204
drwxr-xr-x 2 root   root   20480 Nov  7 22:15 .
drwxrwxr-x 6 jlebon jlebon  4096 Nov  8 16:14 ..
lrwxrwxrwx 1 root   root      29 Nov  8 16:14 /etc/pki/rpm-gpg/RPM-GPG-KEY-31-fedora -> RPM-GPG-KEY-fedora-31-primary
lrwxrwxrwx 1 root   root      29 Nov  8 16:14 /etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-10-i386 -> RPM-GPG-KEY-fedora-10-primary
lrwxrwxrwx 1 root   root      29 Nov  8 16:14 /etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-10-ppc -> RPM-GPG-KEY-fedora-10-primary
lrwxrwxrwx 1 root   root      29 Nov  8 16:14 /etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-10-ppc64 -> RPM-GPG-KEY-fedora-10-primary
lrwxrwxrwx 1 root   root      29 Nov  8 16:14 /etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-10-x86_64 -> RPM-GPG-KEY-fedora-10-primary
...
lrwxrwxrwx 1 root   root      29 Nov  8 16:14 RPM-GPG-KEY-31-fedora -> RPM-GPG-KEY-fedora-31-primary
lrwxrwxrwx 1 root   root      29 Nov  8 16:14 RPM-GPG-KEY-fedora-10-i386 -> RPM-GPG-KEY-fedora-10-primary
lrwxrwxrwx 1 root   root      29 Nov  8 16:14 RPM-GPG-KEY-fedora-10-ppc -> RPM-GPG-KEY-fedora-10-primary
lrwxrwxrwx 1 root   root      29 Nov  8 16:14 RPM-GPG-KEY-fedora-10-ppc64 -> RPM-GPG-KEY-fedora-10-primary
-rw-r--r-- 1 root   root    2380 Oct 11 18:50 RPM-GPG-KEY-fedora-10-primary
lrwxrwxrwx 1 root   root      29 Nov  8 16:14 RPM-GPG-KEY-fedora-10-x86_64 -> RPM-GPG-KEY-fedora-10-primary
...
bash-5.0# ls -la /etc/pki/rpm-gpg | wc -l
384
```

Running e2fsck on `root` complains about this too:

```
Entry '/etc/pki/rpm-gpg/RPM-GPG-KEY-31-fedora' in /etc/pki/rpm-gpg (355) has illegal characters in its name.
Fix<y>? no
Entry '/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-10-i386' in /etc/pki/rpm-gpg (355) has illegal characters in its name.
Fix<y>? no
Entry '/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-10-ppc' in /etc/pki/rpm-gpg (355) has illegal characters in its name.
Fix<y>? no
Entry '/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-10-ppc64' in /etc/pki/rpm-gpg (355) has illegal characters in its name.
Fix<y>? no
Entry '/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-10-x86_64' in /etc/pki/rpm-gpg (355) has illegal characters in its name.
Fix<y>? no
...
```

So I think what's happening there is that before 5.3.8, the kernel tolerated these errors, but now it doesn't.

Comment 1 Richard W.M. Jones 2019-11-08 16:59:16 UTC
We've actually had issues with symlinks before, see bug 1470157
and https://www.redhat.com/archives/libguestfs/2017-July/msg00078.html

However as a result of that patch we are now using the ext2fs_symlink
library function from e2fsprogs, which I hope always creates symlinks
correctly (unless e2fsprogs has a bug).  Eric - any ideas?

Comment 2 Colin Walters 2019-11-08 17:07:12 UTC
While ext4 has a library to make filesystems from userspace (unlike many other filesystems), it's clearly a 2nd tier thing.

Is there a reason for supermin to do this at all versus say just booting from a 9p (or https://virtio-fs.gitlab.io/) rootfs?

Comment 3 Richard W.M. Jones 2019-11-08 17:18:45 UTC
9p isn't in RHEL and virtio-fs didn't exist.  Note however that
mke2fs itself uses the same library to build filesystems (using
the mke2fs -d option) so it's not a second tier method.

Comment 4 Colin Walters 2019-11-08 18:05:18 UTC
> 9p isn't in RHEL and virtio-fs didn't exist.  

Yeah =(

I am really, really looking forward to virtio-fs landing.

> Note however that mke2fs itself uses the same library to build filesystems (using the mke2fs -d option) 

There's a difference between making a filesystem "structure" (superblocks etc.) versus actually writing *content*.
It's the latter that I think is "2nd tier" - bigger picture, most people use the Linux kernel itself to write filesystem content; this is obviously what libguestfs is designed to do, we're just using this ext2 image to bootstrap.
I think OpenEmbedded also uses the ext4 lib, but personally I'm way happier with the flexibility of just requiring KVM.

Comment 5 Eric Sandeen 2019-11-08 18:42:03 UTC
cc: Lukas too but TBH I'm lacking 99% of the context I probably need to have a clue here, as I have no idea what supermin does or how it does it...

Comment 6 Richard W.M. Jones 2019-11-08 18:50:56 UTC
(In reply to Eric Sandeen from comment #5)
> cc: Lukas too but TBH I'm lacking 99% of the context I probably need to have
> a clue here, as I have no idea what supermin does or how it does it...

It's building a small ext4 filesystem in a file to use as a bootable
appliance.  It uses e2fsprogs for this.  There's some issue about whether
symlinks are being created correctly, since kernel 5.3.8 apparently cannot
read them (< 5.3.8 apparently could read them fine).  A simple example
which doesn't need root is:

http://libguestfs.org/supermin.1.html#EXAMPLE

Note I've not investigated this bug at all so far because I've just got
back from 2 weeks of conferences.

(In reply to Colin Walters from comment #4)
> There's a difference between making a filesystem "structure" (superblocks
> etc.) versus actually writing *content*.

mke2fs -d really builds a full filesystem with content.

Comment 7 Toolybird 2019-11-26 00:44:28 UTC
Just hit this same issue on Arch Linux.

supermin ext2 images have always seemed a bit weird to me.

For example, this is on an older LTS kernel:

$ cd /var/tmp
$ wget http://download.libguestfs.org/binaries/appliance/appliance-1.40.1.tar.xz
$ tar -xf appliance-1.40.1.tar.xz
$ sudo mount -o loop appliance/root /mnt/tmp
$ ls -l /mnt/tmp
total 84
lrwxrwxrwx  1 root root     7 Nov 14 03:23 /bin -> usr/bin
lrwxrwxrwx  1 root root     7 Mar  4  2019 bin -> usr/bin
dr-xr-xr-x  2 root root  4096 Mar  4  2019 boot
drwxr-xr-x  2 root root  4096 Mar  4  2019 dev
drwxr-xr-x 58 gws  gws   4096 Jan 18  2019 etc
drwxr-xr-x  2 root root  4096 Dec 13  2018 home
-rwxr-xr-x  1 gws  gws   7120 Jan 17  2019 init
lrwxrwxrwx  1 root root     7 Nov 14 03:23 /lib -> usr/lib
lrwxrwxrwx  1 root root     7 Mar  4  2019 lib -> usr/lib
lrwxrwxrwx  1 root root     7 Nov 14 03:23 /lib64 -> usr/lib
lrwxrwxrwx  1 root root     9 Mar  4  2019 lib64 -> usr/lib64
drwx------  2 root root 16384 Mar  4  2019 lost+found
drwxr-xr-x  2 root root  4096 Jul 13  2018 media
drwxr-xr-x  2 root root  4096 Dec 14  2018 mnt
drwxr-xr-x  2 root root  4096 Jul 13  2018 opt
dr-xr-xr-x  2 root root  4096 Feb  4  2019 proc
dr-xr-x---  2 root root  4096 Jan 12  2019 root
drwxr-xr-x 12 root root  4096 Mar  4  2019 run
lrwxrwxrwx  1 root root     7 Nov 14 03:23 /sbin -> usr/bin
lrwxrwxrwx  1 root root     8 Mar  4  2019 sbin -> usr/sbin
drwxr-xr-x  2 root root  4096 Jul 13  2018 srv
dr-xr-xr-x  2 root root  4096 Feb  4  2019 sys
drwxrwxrwt  2 root root  4096 Mar  4  2019 tmp
drwxr-xr-x 12 gws  gws   4096 Jan 18  2019 usr
drwxr-xr-x 19 root root  4096 Feb  1  2019 var

Notice those '/' characters (forward slashes) in the directory entries? They just look wrong.

I'd guess they are the "illegal characters" and the source of the problem.

Comment 8 Toolybird 2019-11-26 01:53:40 UTC
I haven't bisected it to confirm, but this commit appeared in 5.3.8 and looks relevant:

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/fs/readdir.c?id=8a23eb804ca4f2be909e372cf5a9e7b30ae476cd

Comment 9 Richard W.M. Jones 2019-11-26 08:44:28 UTC
Forgotten about this bug, thanks for reminding me.  Yes I'm able to reproduce it
easily with kernel 5.4.0.  Here's a small reproducer:

$ supermin --prepare bash util-linux -o /tmp/supermin.d
$ supermin --build /tmp/supermin.d -f ext2 -o /tmp/appliance.d
$ ll /tmp/appliance.d/root 
-rw-r--r--. 1 rjones rjones 4294967296 Nov 26 08:38 /tmp/appliance.d/root
$ guestfish --ro -a /tmp/appliance.d/root -m /dev/sda ll /
libguestfs: error: ll: ls: reading directory '/sysroot/': Input/output error

Copying the /tmp/appliance.d/root file to a < 5.3.8 kernel machine shows:

lrwxrwxrwx  1 root root     7 Nov 26 08:43 /bin -> usr/bin
lrwxrwxrwx  1 root root     7 Nov 26 08:43 /lib -> usr/lib
lrwxrwxrwx  1 root root     9 Nov 26 08:43 /lib64 -> usr/lib64
lrwxrwxrwx  1 root root     8 Nov 26 08:43 /sbin -> usr/sbin
lrwxrwxrwx  1 root root     7 Nov 26 08:38 bin -> usr/bin

which are clearly invalid directory entries.

It does indeed look as if it's a bug in supermin.  Also the patch in comment 8
does look very relevant.

Comment 10 Richard W.M. Jones 2019-11-26 09:02:29 UTC
Created attachment 1639725 [details]
0001-ext2-Build-symbolic-links-correctly-RHBZ-1770304.patch

This is only lightly tested but works for me so far.  I will
do a bit more testing this morning and post it formally on the
mailing list if it works.

Comment 11 Richard W.M. Jones 2019-11-26 09:11:00 UTC
Patch posted:
https://www.redhat.com/archives/libguestfs/2019-November/msg00262.html

Comment 12 Toolybird 2019-11-27 01:21:38 UTC
Patch works for me. Thanks!

Comment 13 Ben Cotton 2019-11-27 14:23:35 UTC
Fedora 29 changed to end-of-life (EOL) status on 2019-11-26. Fedora 29 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.

Comment 14 Jonathan Lebon 2019-11-27 14:27:10 UTC
This bug is opened against f31, not f29.

Comment 16 Fedora Update System 2019-11-28 14:03:34 UTC
FEDORA-2019-f4bf176b5c has been submitted as an update to Fedora 30. https://bodhi.fedoraproject.org/updates/FEDORA-2019-f4bf176b5c

Comment 17 Fedora Update System 2019-11-28 14:03:40 UTC
FEDORA-2019-e8415f65c9 has been submitted as an update to Fedora 31. https://bodhi.fedoraproject.org/updates/FEDORA-2019-e8415f65c9

Comment 18 Fedora Update System 2019-11-29 01:20:43 UTC
supermin-5.1.20-10.fc30 has been pushed to the Fedora 30 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-f4bf176b5c

Comment 19 Fedora Update System 2019-11-29 01:30:55 UTC
supermin-5.1.20-10.fc31 has been pushed to the Fedora 31 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-e8415f65c9

Comment 20 Richard W.M. Jones 2019-12-10 15:52:56 UTC
See also https://bugzilla.redhat.com/show_bug.cgi?id=1781803

Comment 21 Richard W.M. Jones 2019-12-10 18:50:20 UTC
*** Bug 1781803 has been marked as a duplicate of this bug. ***

Comment 22 Fedora Update System 2019-12-10 18:57:33 UTC
FEDORA-2019-3454e38e8c has been submitted as an update to Fedora 30. https://bodhi.fedoraproject.org/updates/FEDORA-2019-3454e38e8c

Comment 23 Fedora Update System 2019-12-10 19:03:59 UTC
FEDORA-2019-0fb47d9231 has been submitted as an update to Fedora 31. https://bodhi.fedoraproject.org/updates/FEDORA-2019-0fb47d9231

Comment 24 Fedora Update System 2019-12-11 01:04:23 UTC
supermin-5.1.20-11.fc30 has been pushed to the Fedora 30 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-3454e38e8c

Comment 25 Fedora Update System 2019-12-11 01:41:56 UTC
supermin-5.1.20-11.fc31 has been pushed to the Fedora 31 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-0fb47d9231

Comment 26 Fedora Update System 2020-01-03 20:35:34 UTC
supermin-5.1.20-11.fc30 has been pushed to the Fedora 30 stable repository. If problems still persist, please make note of it in this bug report.

Comment 27 Fedora Update System 2020-01-05 00:39:28 UTC
supermin-5.1.20-11.fc31 has been pushed to the Fedora 31 stable repository. If problems still persist, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.