Bug 1770304

Summary: ext2 filesystem supermin assembles doesn't write symlinks correctly
Product: [Fedora] Fedora Reporter: Jonathan Lebon <jlebon>
Component: superminAssignee: Richard W.M. Jones <rjones>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 31CC: dustymabe, esandeen, lczerner, miabbott, ptoscano, rjones, toolybird, walters
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: supermin-5.1.20-11.fc30 supermin-5.1.20-11.fc31 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1776719 (view as bug list) Environment:
Last Closed: 2020-01-03 20:35:34 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1776719    
Attachments:
Description Flags
0001-ext2-Build-symbolic-links-correctly-RHBZ-1770304.patch none

Description Jonathan Lebon 2019-11-08 16:48:35 UTC
Description of problem:

I initially was going to file this against the kernel, but I'm leaning more now towards supermin doing something funky. After upgrading the kernel from 5.3.7 to 5.3.8, we've noticed a regression where our supermin appliance only sees one file in `/etc/pki/rpm-gpg`, instead of all the GPG keys.

Version-Release number of selected component (if applicable):

Reproduces with supermin-5.1.20-8.fc31.x86_64 + kernel-5.3.[89].
Does not reproduce with supermin-5.1.20-8.fc31.x86_64 + kernel-5.3.7.

How reproducible:

100%

Steps to Reproduce:

```
$ export SUPERMIN_KERNEL=/usr/lib/modules/5.3.8-300.fc31.x86_64/vmlinuz
$ export SUPERMIN_MODULES=/usr/lib/modules/5.3.8-300.fc31.x86_64
$ rpm -q fedora-gpg-keys
fedora-gpg-keys-31-1.noarch
$ ls -la /etc/pki/rpm-gpg | wc -l
218
$ supermin --prepare --use-installed -o supermin.prepare bash coreutils fedora-gpg-keys util-linux
$ tar zcf supermin.prepare/init.tar.gz ./init
$ supermin --build supermin.prepare --size 5G -f ext2 -o supermin.build
$ qemu-kvm -nodefaults -nographic -kernel supermin.build/kernel -initrd supermin.build/initrd -hda supermin.build/root -serial stdio -append "console=ttyS0 root=/dev/sda"
...
bash-5.0# ls -la /etc/pki/rpm-gpg
ls: reading directory '/etc/pki/rpm-gpg': Input/output error
total 24
drwxr-xr-x 2 root   root   20480 Nov  7 22:15 .
drwxrwxr-x 6 jlebon jlebon  4096 Nov  8 16:07 ..
lrwxrwxrwx 1 root   root      29 Nov  8 16:07 RPM-GPG-KEY-31-fedora -> RPM-GPG-KEY-fedora-31-primary
bash-5.0#
```

Actual results:

Only one broken symlink.

Expected results:

All the GPG keys + symlinks.

Additional info:

What's interesting is that running with 5.3.7, all the files show up, but it seems like symlinks show up twice: once with the absolute path, and once without:

```
bash-5.0# cd /etc/pki/rpm-gpg
bash-5.0# ls -la
total 204
drwxr-xr-x 2 root   root   20480 Nov  7 22:15 .
drwxrwxr-x 6 jlebon jlebon  4096 Nov  8 16:14 ..
lrwxrwxrwx 1 root   root      29 Nov  8 16:14 /etc/pki/rpm-gpg/RPM-GPG-KEY-31-fedora -> RPM-GPG-KEY-fedora-31-primary
lrwxrwxrwx 1 root   root      29 Nov  8 16:14 /etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-10-i386 -> RPM-GPG-KEY-fedora-10-primary
lrwxrwxrwx 1 root   root      29 Nov  8 16:14 /etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-10-ppc -> RPM-GPG-KEY-fedora-10-primary
lrwxrwxrwx 1 root   root      29 Nov  8 16:14 /etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-10-ppc64 -> RPM-GPG-KEY-fedora-10-primary
lrwxrwxrwx 1 root   root      29 Nov  8 16:14 /etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-10-x86_64 -> RPM-GPG-KEY-fedora-10-primary
...
lrwxrwxrwx 1 root   root      29 Nov  8 16:14 RPM-GPG-KEY-31-fedora -> RPM-GPG-KEY-fedora-31-primary
lrwxrwxrwx 1 root   root      29 Nov  8 16:14 RPM-GPG-KEY-fedora-10-i386 -> RPM-GPG-KEY-fedora-10-primary
lrwxrwxrwx 1 root   root      29 Nov  8 16:14 RPM-GPG-KEY-fedora-10-ppc -> RPM-GPG-KEY-fedora-10-primary
lrwxrwxrwx 1 root   root      29 Nov  8 16:14 RPM-GPG-KEY-fedora-10-ppc64 -> RPM-GPG-KEY-fedora-10-primary
-rw-r--r-- 1 root   root    2380 Oct 11 18:50 RPM-GPG-KEY-fedora-10-primary
lrwxrwxrwx 1 root   root      29 Nov  8 16:14 RPM-GPG-KEY-fedora-10-x86_64 -> RPM-GPG-KEY-fedora-10-primary
...
bash-5.0# ls -la /etc/pki/rpm-gpg | wc -l
384
```

Running e2fsck on `root` complains about this too:

```
Entry '/etc/pki/rpm-gpg/RPM-GPG-KEY-31-fedora' in /etc/pki/rpm-gpg (355) has illegal characters in its name.
Fix<y>? no
Entry '/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-10-i386' in /etc/pki/rpm-gpg (355) has illegal characters in its name.
Fix<y>? no
Entry '/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-10-ppc' in /etc/pki/rpm-gpg (355) has illegal characters in its name.
Fix<y>? no
Entry '/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-10-ppc64' in /etc/pki/rpm-gpg (355) has illegal characters in its name.
Fix<y>? no
Entry '/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-10-x86_64' in /etc/pki/rpm-gpg (355) has illegal characters in its name.
Fix<y>? no
...
```

So I think what's happening there is that before 5.3.8, the kernel tolerated these errors, but now it doesn't.

Comment 1 Richard W.M. Jones 2019-11-08 16:59:16 UTC
We've actually had issues with symlinks before, see bug 1470157
and https://www.redhat.com/archives/libguestfs/2017-July/msg00078.html

However as a result of that patch we are now using the ext2fs_symlink
library function from e2fsprogs, which I hope always creates symlinks
correctly (unless e2fsprogs has a bug).  Eric - any ideas?

Comment 2 Colin Walters 2019-11-08 17:07:12 UTC
While ext4 has a library to make filesystems from userspace (unlike many other filesystems), it's clearly a 2nd tier thing.

Is there a reason for supermin to do this at all versus say just booting from a 9p (or https://virtio-fs.gitlab.io/) rootfs?

Comment 3 Richard W.M. Jones 2019-11-08 17:18:45 UTC
9p isn't in RHEL and virtio-fs didn't exist.  Note however that
mke2fs itself uses the same library to build filesystems (using
the mke2fs -d option) so it's not a second tier method.

Comment 4 Colin Walters 2019-11-08 18:05:18 UTC
> 9p isn't in RHEL and virtio-fs didn't exist.  

Yeah =(

I am really, really looking forward to virtio-fs landing.

> Note however that mke2fs itself uses the same library to build filesystems (using the mke2fs -d option) 

There's a difference between making a filesystem "structure" (superblocks etc.) versus actually writing *content*.
It's the latter that I think is "2nd tier" - bigger picture, most people use the Linux kernel itself to write filesystem content; this is obviously what libguestfs is designed to do, we're just using this ext2 image to bootstrap.
I think OpenEmbedded also uses the ext4 lib, but personally I'm way happier with the flexibility of just requiring KVM.

Comment 5 Eric Sandeen 2019-11-08 18:42:03 UTC
cc: Lukas too but TBH I'm lacking 99% of the context I probably need to have a clue here, as I have no idea what supermin does or how it does it...

Comment 6 Richard W.M. Jones 2019-11-08 18:50:56 UTC
(In reply to Eric Sandeen from comment #5)
> cc: Lukas too but TBH I'm lacking 99% of the context I probably need to have
> a clue here, as I have no idea what supermin does or how it does it...

It's building a small ext4 filesystem in a file to use as a bootable
appliance.  It uses e2fsprogs for this.  There's some issue about whether
symlinks are being created correctly, since kernel 5.3.8 apparently cannot
read them (< 5.3.8 apparently could read them fine).  A simple example
which doesn't need root is:

http://libguestfs.org/supermin.1.html#EXAMPLE

Note I've not investigated this bug at all so far because I've just got
back from 2 weeks of conferences.

(In reply to Colin Walters from comment #4)
> There's a difference between making a filesystem "structure" (superblocks
> etc.) versus actually writing *content*.

mke2fs -d really builds a full filesystem with content.

Comment 7 Toolybird 2019-11-26 00:44:28 UTC
Just hit this same issue on Arch Linux.

supermin ext2 images have always seemed a bit weird to me.

For example, this is on an older LTS kernel:

$ cd /var/tmp
$ wget http://download.libguestfs.org/binaries/appliance/appliance-1.40.1.tar.xz
$ tar -xf appliance-1.40.1.tar.xz
$ sudo mount -o loop appliance/root /mnt/tmp
$ ls -l /mnt/tmp
total 84
lrwxrwxrwx  1 root root     7 Nov 14 03:23 /bin -> usr/bin
lrwxrwxrwx  1 root root     7 Mar  4  2019 bin -> usr/bin
dr-xr-xr-x  2 root root  4096 Mar  4  2019 boot
drwxr-xr-x  2 root root  4096 Mar  4  2019 dev
drwxr-xr-x 58 gws  gws   4096 Jan 18  2019 etc
drwxr-xr-x  2 root root  4096 Dec 13  2018 home
-rwxr-xr-x  1 gws  gws   7120 Jan 17  2019 init
lrwxrwxrwx  1 root root     7 Nov 14 03:23 /lib -> usr/lib
lrwxrwxrwx  1 root root     7 Mar  4  2019 lib -> usr/lib
lrwxrwxrwx  1 root root     7 Nov 14 03:23 /lib64 -> usr/lib
lrwxrwxrwx  1 root root     9 Mar  4  2019 lib64 -> usr/lib64
drwx------  2 root root 16384 Mar  4  2019 lost+found
drwxr-xr-x  2 root root  4096 Jul 13  2018 media
drwxr-xr-x  2 root root  4096 Dec 14  2018 mnt
drwxr-xr-x  2 root root  4096 Jul 13  2018 opt
dr-xr-xr-x  2 root root  4096 Feb  4  2019 proc
dr-xr-x---  2 root root  4096 Jan 12  2019 root
drwxr-xr-x 12 root root  4096 Mar  4  2019 run
lrwxrwxrwx  1 root root     7 Nov 14 03:23 /sbin -> usr/bin
lrwxrwxrwx  1 root root     8 Mar  4  2019 sbin -> usr/sbin
drwxr-xr-x  2 root root  4096 Jul 13  2018 srv
dr-xr-xr-x  2 root root  4096 Feb  4  2019 sys
drwxrwxrwt  2 root root  4096 Mar  4  2019 tmp
drwxr-xr-x 12 gws  gws   4096 Jan 18  2019 usr
drwxr-xr-x 19 root root  4096 Feb  1  2019 var

Notice those '/' characters (forward slashes) in the directory entries? They just look wrong.

I'd guess they are the "illegal characters" and the source of the problem.

Comment 8 Toolybird 2019-11-26 01:53:40 UTC
I haven't bisected it to confirm, but this commit appeared in 5.3.8 and looks relevant:

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/fs/readdir.c?id=8a23eb804ca4f2be909e372cf5a9e7b30ae476cd

Comment 9 Richard W.M. Jones 2019-11-26 08:44:28 UTC
Forgotten about this bug, thanks for reminding me.  Yes I'm able to reproduce it
easily with kernel 5.4.0.  Here's a small reproducer:

$ supermin --prepare bash util-linux -o /tmp/supermin.d
$ supermin --build /tmp/supermin.d -f ext2 -o /tmp/appliance.d
$ ll /tmp/appliance.d/root 
-rw-r--r--. 1 rjones rjones 4294967296 Nov 26 08:38 /tmp/appliance.d/root
$ guestfish --ro -a /tmp/appliance.d/root -m /dev/sda ll /
libguestfs: error: ll: ls: reading directory '/sysroot/': Input/output error

Copying the /tmp/appliance.d/root file to a < 5.3.8 kernel machine shows:

lrwxrwxrwx  1 root root     7 Nov 26 08:43 /bin -> usr/bin
lrwxrwxrwx  1 root root     7 Nov 26 08:43 /lib -> usr/lib
lrwxrwxrwx  1 root root     9 Nov 26 08:43 /lib64 -> usr/lib64
lrwxrwxrwx  1 root root     8 Nov 26 08:43 /sbin -> usr/sbin
lrwxrwxrwx  1 root root     7 Nov 26 08:38 bin -> usr/bin

which are clearly invalid directory entries.

It does indeed look as if it's a bug in supermin.  Also the patch in comment 8
does look very relevant.

Comment 10 Richard W.M. Jones 2019-11-26 09:02:29 UTC
Created attachment 1639725 [details]
0001-ext2-Build-symbolic-links-correctly-RHBZ-1770304.patch

This is only lightly tested but works for me so far.  I will
do a bit more testing this morning and post it formally on the
mailing list if it works.

Comment 11 Richard W.M. Jones 2019-11-26 09:11:00 UTC
Patch posted:
https://www.redhat.com/archives/libguestfs/2019-November/msg00262.html

Comment 12 Toolybird 2019-11-27 01:21:38 UTC
Patch works for me. Thanks!

Comment 13 Ben Cotton 2019-11-27 14:23:35 UTC
Fedora 29 changed to end-of-life (EOL) status on 2019-11-26. Fedora 29 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.

Comment 14 Jonathan Lebon 2019-11-27 14:27:10 UTC
This bug is opened against f31, not f29.

Comment 16 Fedora Update System 2019-11-28 14:03:34 UTC
FEDORA-2019-f4bf176b5c has been submitted as an update to Fedora 30. https://bodhi.fedoraproject.org/updates/FEDORA-2019-f4bf176b5c

Comment 17 Fedora Update System 2019-11-28 14:03:40 UTC
FEDORA-2019-e8415f65c9 has been submitted as an update to Fedora 31. https://bodhi.fedoraproject.org/updates/FEDORA-2019-e8415f65c9

Comment 18 Fedora Update System 2019-11-29 01:20:43 UTC
supermin-5.1.20-10.fc30 has been pushed to the Fedora 30 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-f4bf176b5c

Comment 19 Fedora Update System 2019-11-29 01:30:55 UTC
supermin-5.1.20-10.fc31 has been pushed to the Fedora 31 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-e8415f65c9

Comment 20 Richard W.M. Jones 2019-12-10 15:52:56 UTC
See also https://bugzilla.redhat.com/show_bug.cgi?id=1781803

Comment 21 Richard W.M. Jones 2019-12-10 18:50:20 UTC
*** Bug 1781803 has been marked as a duplicate of this bug. ***

Comment 22 Fedora Update System 2019-12-10 18:57:33 UTC
FEDORA-2019-3454e38e8c has been submitted as an update to Fedora 30. https://bodhi.fedoraproject.org/updates/FEDORA-2019-3454e38e8c

Comment 23 Fedora Update System 2019-12-10 19:03:59 UTC
FEDORA-2019-0fb47d9231 has been submitted as an update to Fedora 31. https://bodhi.fedoraproject.org/updates/FEDORA-2019-0fb47d9231

Comment 24 Fedora Update System 2019-12-11 01:04:23 UTC
supermin-5.1.20-11.fc30 has been pushed to the Fedora 30 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-3454e38e8c

Comment 25 Fedora Update System 2019-12-11 01:41:56 UTC
supermin-5.1.20-11.fc31 has been pushed to the Fedora 31 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-0fb47d9231

Comment 26 Fedora Update System 2020-01-03 20:35:34 UTC
supermin-5.1.20-11.fc30 has been pushed to the Fedora 30 stable repository. If problems still persist, please make note of it in this bug report.

Comment 27 Fedora Update System 2020-01-05 00:39:28 UTC
supermin-5.1.20-11.fc31 has been pushed to the Fedora 31 stable repository. If problems still persist, please make note of it in this bug report.