Bug 1855034

Summary: Creating an armhfp disk image with btrfs results in an empty filesystem
Product: [Fedora] Fedora Reporter: Paul Whalen <pwhalen>
Component: appliance-toolsAssignee: Neal Gompa <ngompa13>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 33CC: agrimm, awilliam, bugzilla, davide, dhuff, josef, michel, ngompa13, pbrobinson, robatino
Target Milestone: ---Keywords: Reopened
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: AcceptedFreezeException
Fixed In Version: appliance-tools-011.0-1.fc31 appliance-tools-011.0-1.fc32 appliance-tools-011.0-1.el8 appliance-tools-011.1-1.fc33 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-08-28 02:38:03 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1766776    
Attachments:
Description Flags
Kickstart
none
appliance-creator-log none

Description Paul Whalen 2020-07-08 17:40:19 UTC
Description of problem:

Attempted to create an armhfp disk image for BtrfsByDefault change proposed in Fedora 33, the task completed successfully but resulted in an empty rootfs. 


Version-Release number of selected component (if applicable):
appliance-tools-009.0-7.fc31

How reproducible:


Steps to Reproduce:
Command used:
/usr/bin/appliance-creator -c Fedora-Xfce-armhfp-Rawhide-20200708.n.0-btrfs.ks -d -v --logfile appliance.log --cache /tmp/koji-appliance -o app-output --format raw --name Fedora-Xfce-armhfp-Rawhide-20200708 --version Rawhide --release 20200708.n.0


Actual results:
No errors, but the rootfs is empty.

Expected results:


Additional info:

Comment 1 Paul Whalen 2020-07-08 17:42:11 UTC
Created attachment 1700339 [details]
Kickstart

Comment 2 Paul Whalen 2020-07-08 17:45:12 UTC
Created attachment 1700340 [details]
appliance-creator-log

Comment 3 Chris Murphy 2020-07-08 18:33:24 UTC
Is it possible to get more verbose logs? A debugging mode perhaps?

Mounting /dev/loop03 at /var/tmp/imgcreate-qpevrvqw/install_rootbtrfs.007


This doesn't say what mount option is used, if any.

Comment 4 Paul Whalen 2020-07-08 19:13:42 UTC
The disk gets mounted at "/var/tmp/imgcreate-fbc558xm/install_rootbtrfs.007" but the packages are installed to "/var/tmp/imgcreate-fbc558xm/install_root/"

/dev/mapper/loop0p2 on /var/tmp/imgcreate-fbc558xm/install_root/boot type ext4 (rw,relatime,seclabel)
/dev/mapper/loop0p1 on /var/tmp/imgcreate-fbc558xm/install_root/boot/efi type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,errors=remount-ro)
/dev/mapper/loop0p3 on /var/tmp/imgcreate-fbc558xm/install_rootbtrfs.007 type btrfs (rw,relatime,seclabel,space_cache,subvolid=5,subvol=/)

[root@builder install_root]# pwd
/var/tmp/imgcreate-fbc558xm/install_root
[root@builder install_root]# ls
bin  boot  dev  etc  home  lib  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var

Comment 5 Davide Cavalca 2020-07-08 23:27:02 UTC
From a quick look, the partitioning logic at https://pagure.io/appliance-tools/blob/2265560b4bca24e19452a34ab8e0acf235aabfa0/f/appcreate/partitionedfs.py#_113 might need to be extended to cover btrfs as well.

Comment 6 Davide Cavalca 2020-07-08 23:57:51 UTC
https://pagure.io/appliance-tools/blob/2265560b4bca24e19452a34ab8e0acf235aabfa0/f/appcreate/appliance.py#_384 also needs to be extended to pass rootflags via extlinux. There's a similar logic at https://pagure.io/appliance-tools/blob/2265560b4bca24e19452a34ab8e0acf235aabfa0/f/appcreate/appliance.py#_324 for grub-legacy, but I'm not sure if it's still in use.

Comment 7 Neal Gompa 2020-07-09 22:56:29 UTC
Nobody better be using grub legacy anymore. :)

If we can rip that out as part of this, I think nobody would bat an eye.

Comment 8 Chris Murphy 2020-07-09 23:25:20 UTC
GRUB2 was Fedora 15, May 2011.
https://fedoraproject.org/wiki/Releases/16/FeatureList

Abandon hope all ye who enter here. Gut the legacy with reckless abandon, let mayhem be avoided, and sanity reign supreme.

Comment 9 Neal Gompa 2020-07-10 02:03:44 UTC
So here's a trivial way to run this and attempt to reproduce it:

$ sudo dnf install mock qemu-user-static wget
$ sudo usermod -a -G mock $USER
$ newgrp mock
$ wget --content-disposition "https://bugzilla.redhat.com/attachment.cgi?id=1700339"
$ mock --root fedora-rawhide-armhfp --install appliance-tools btrfs-progs
$ mock --root fedora-rawhide-armhfp --copyin Fedora-Xfce-armhfp-Rawhide-20200708.n.0-btrfs.ks /
$ mock --root fedora-rawhide-armhfp --isolation=simple --chroot "/usr/bin/appliance-creator -c /Fedora-Xfce-armhfp-Rawhide-20200708.n.0-btrfs.ks -d -v --logfile appliance.log --cache /tmp/koji-appliance -o app-output --format raw --name Fedora-Xfce-armhfp-Rawhide-20200708 --version Rawhide --release 20200708.n.0"

Comment 10 Davide Cavalca 2020-07-10 05:14:04 UTC
So the problem is that for some reason this is writing stuff out into /var/tmp/imgcreate-5kvkhew1/install_root, while the root volume for the image is mounted on /var/tmp/imgcreate-5kvkhew1/install_rootbtrfs.007 and hence remains empty.

Comment 11 Davide Cavalca 2020-07-10 05:21:45 UTC
Put up https://pagure.io/appliance-tools/pull-request/10 for a minor appliance-tools fix. Note that this doesn't actually resolve the issue, which I think might be in imgcreate (as that's where the install_root path bit is defined): https://github.com/livecd-tools/livecd-tools/tree/master/imgcreate

Comment 12 Davide Cavalca 2020-07-10 05:42:53 UTC
Ok so the *actual* problem is that this thing doesn't understand subvolumes at all. This will require changes in both imgcreate (to add subvolume support) and to appliance-tools (to parse the ks correctly and plumb them in). https://pagure.io/appliance-tools/blob/2265560b4bca24e19452a34ab8e0acf235aabfa0/f/appcreate/appliance.py#_224 is the entry point, the parsing happens at the beginning of that function.

Comment 13 Davide Cavalca 2020-07-10 16:19:11 UTC
Tentative fix to mitigate this: https://pagure.io/appliance-tools/pull-request/11

Comment 14 Davide Cavalca 2020-07-10 18:50:56 UTC
And https://pagure.io/appliance-tools/pull-request/12 should take care of rootflags in the bootloader

Comment 15 Neal Gompa 2020-07-10 19:11:14 UTC
Released appliance-tools-010.0-1.fc33 to Rawhide, which incorporates those fixes: https://koji.fedoraproject.org/koji/buildinfo?buildID=1539885

@Paul, could you test this with the new appliance-creator?

Comment 16 Neal Gompa 2020-07-11 00:01:02 UTC
Well, apparently that didn't work, at least based on this: https://koji.fedoraproject.org/koji/taskinfo?taskID=46963623

Comment 17 Davide Cavalca 2020-07-11 00:34:08 UTC
Error looks like "Unable to create appliance : Failed to unmap partitions for '/dev/loop1'", which points to the unmount logic: https://pagure.io/appliance-tools/blob/7e39c8e04f87d8ad501e55dac016472b2ca445e0/f/appcreate/partitionedfs.py#_246

Comment 18 Davide Cavalca 2020-07-11 00:36:31 UTC
Interestingly, this ran fine when I tried it here (on Rawhide, using Neal's repro), so it could also be some interplay with the koji environment.

Comment 19 Neal Gompa 2020-07-11 21:58:57 UTC
It might be worth comparing how the logs look for Xfce vs Server:

* Xfce: https://koji.fedoraproject.org/koji/taskinfo?taskID=46963623
* Server: https://koji.fedoraproject.org/koji/taskinfo?taskID=46963570

At least the mountpoint stuff looks fishy, but I'm not sure...

Unfortunately, it seems like the packager dev/test arm machines[1] are still down...

[1]: https://fedoraproject.org/wiki/Test_Machine_Resources_For_Package_Maintainers

Comment 20 Chris Murphy 2020-07-11 23:21:02 UTC
Three differences I notice: (1) Xfce mounts root after /boot and /boot/efi (2) Xfce mount root at a different location than /boot and /boot/efi - either of these is fatal. (3) Xfce shows the mounted loop devices are loop11 loop12 loop13 which coorespond to device mapper devices; Server shows loop devices are loop01 loop02 loop03 also dm targets. But where are loop01->loop10 on Xfce? These getting setup could explain the delay for loop13 which is the one for the btrfs root, that gets mounted late.

Is there supposed to be a 'root' subvolume on this Btrfs, similar to Anaconda installations? If so, seems like there needs to be: mkfs.btrfs; mount it; btrfs sub create root; umount; mount -o subvol=root. I don't see evidence it's mounted twice *and* before /boot or /boot/efi.

Also check out line 13.

Xfce:
Assigned btrfs.007 to sda3 at 596 at size 5000
Server:
Assigned / to sda3 at 596 at size 2500


Assigned btrfs.007 seems wrong, and ends up being appended to the mountpoint we want:

Server:
Mounting /dev/loop03 at /var/tmp/imgcreate-9v0ntu3p/install_root/
Xfce:
Mounting /dev/loop13 at /var/tmp/imgcreate-dj6hn4s7/install_rootbtrfs.007

(A minor detail is that the sizes seem off. 5000 vs 2500. But it could be intentional and probably isn't related to any of the above.)

Comment 21 Neal Gompa 2020-07-11 23:28:21 UTC
I noticed that Mock was choking on running appliance-creator locally due to lack of btrfs-control device, so sent a PR for that: https://github.com/rpm-software-management/mock/pull/602

Comment 22 Neal Gompa 2020-07-11 23:28:51 UTC
Interestingly, the errors I got locally do not happen in Koji, so something odd is going on there...

Comment 23 Davide Cavalca 2020-07-11 23:30:45 UTC
Another minor appliance-tools fix at https://pagure.io/appliance-tools/pull-request/13. I'm also trying to spin up rawhide on a Raspberry Pi 3 to rule out mock, but haven't managed to get it to boot yet.

Comment 24 Chris Murphy 2020-07-11 23:40:51 UTC
I'm also noticing this difference.

Server
Losetup add /dev/loop0 mapping to /var/tmp/imgcreate-9v0ntu3p/tmp-xtdifk16/Fedora-Server-armhfp-Rawhide-20200710.n.1-sda.raw
Formatting disks
Initializing partition table for /dev/loop0 with msdos layout

Xfce
Losetup add /dev/loop1 mapping to /var/tmp/imgcreate-dj6hn4s7/tmp-vsjy29iu/Fedora-Xfce-armhfp-Rawhide-20200710.n.1-sda.raw
Formatting disks
Initializing partition table for /dev/loop1 with msdos layout

Where is loop0 in the Xfce case? Is something creating a ton of loop devices and that's why the btrfs root is so late to the party?

Comment 25 Neal Gompa 2020-07-13 03:37:27 UTC
So now we have a new, different error with the latest Xfce build: https://koji.fedoraproject.org/koji/taskinfo?taskID=47074262

Koji says that appliance-builder returns an error saying / doesn't exist. But in the appliance log, the image was built?!

Comment 26 Neal Gompa 2020-07-13 03:38:59 UTC
At this point, it seems like we're having trouble just mounting the filesystem correctly so that the dnf installroot operation sets everything up properly...

Comment 27 Davide Cavalca 2020-07-13 03:41:55 UTC
That error comes from koji: https://pagure.io/koji/blob/master/f/builder/kojid#_3184

Looks like it's only looking at partitions in the kickstart, and needs to be extended to understand subvolumes as well.

Comment 28 Davide Cavalca 2020-07-13 03:58:09 UTC
Put up https://pagure.io/koji/pull-request/2365 to fix the Koji issue.

Comment 29 Davide Cavalca 2020-08-09 18:48:51 UTC
Koji was pushed, builds are succeeding now: https://koji.fedoraproject.org/koji/taskinfo?taskID=48954684

Comment 30 Paul Whalen 2020-08-10 20:28:34 UTC
Checking the Xfce disk image from today(Fedora-Rawhide-20200810.n.0)- the rootfs now has the expected subvolumes root and home. However- everything is installed into the root subvolume including the kernel/initrd and firmware files. The first VFAT partition should contain the Raspberry Pi uboot and firmware, the second partition should be used for /boot and have the kernel, initrd and extlinux.conf. Partition 1 is empty, partition 2 has a single empty folder 'efi' (which should be used as the mount point for partition 1). 

Looking at the appliance log from the task in the comment above, the mounts used for partition 1&2 are missing 'btrfs.007':

https://kojipkgs.fedoraproject.org//work/tasks/4684/48954684/appliance.log

Creating mount point /var/tmp/imgcreate-4_juqjb0/install_root/boot
Mounting /dev/loop02 at /var/tmp/imgcreate-4_juqjb0/install_root/boot

Creating mount point /var/tmp/imgcreate-4_juqjb0/install_root/boot/efi
Mounting /dev/loop01 at /var/tmp/imgcreate-4_juqjb0/install_root/boot/efi

Rootfs:
Creating mount point /var/tmp/imgcreate-4_juqjb0/install_rootbtrfs.007
Mounting /dev/loop03 at /var/tmp/imgcreate-4_juqjb0/install_rootbtrfs.007

Comment 31 Chris Murphy 2020-08-10 21:00:05 UTC
Yeah it's still not assembling the pieces in the correct order, if the log order is reliably the order things are happening in.

The following things stand out:

>No --ondisk specified in partition line of ks file; assuming 'sda'

Should it be specified?


>Forcing disk label to msdos, because bootloader is extlinux-bootloader

Why is this msdos and extlinux? when this is supposed to be UEFI with a /boot/efi mount point? This seems like a mismatch.


>Creating mount point /var/tmp/imgcreate-4_juqjb0/install_root/boot
>Mounting /dev/loop02 at /var/tmp/imgcreate-4_juqjb0/install_root/boot

It starts assembling things before it's even completed the mkfs.btrfs or mkdosfs commands. It mount the ext4 boot volume before the root volume is formatted and mounted, which is obviously wrong.


>Formating args: ['mkfs.btrfs', '-L', '_btrfs.007', '/dev/loop03']

Goofy label. Where is this label name coming from? Is it intended for something else?


>Creating mount point /var/tmp/imgcreate-4_juqjb0/install_rootbtrfs.007
>Mounting /dev/loop03 at /var/tmp/imgcreate-4_juqjb0/install_rootbtrfs.007

It still seems confused to me, like it's tacking on 'btrfs.007' in multiple places it doesn't belong: a. label, b. the mount point should just be install_root.


>Writing mkinitrd config /var/tmp/imgcreate-4_juqjb0/install_root/etc/sysconfig/mkinitrd

If the log is to be trusted, this path doesn't exist on btrfs. It very well should be install_root/etc but in fact the mount point created is install_rootbtrfs.007


>Writing extlinux config /var/tmp/imgcreate-4_juqjb0/install_root/boot/extlinux/extlinux.conf

Another path that does not exist on persistent media.

>umount:
>/var/tmp/imgcreate-4_juqjb0/install_root/var/cache/dnf:
>target
>is
>busy.
>Unable to unmount /var/tmp/imgcreate-4_juqjb0/install_root/var/cache/dnf normally, using lazy unmount

This isn't a mount point, of course it can't be umounted. Why does it try to do this?

Comment 32 Peter Robinson 2020-08-10 21:10:28 UTC
> >Forcing disk label to msdos, because bootloader is extlinux-bootloader
> 
> Why is this msdos and extlinux? when this is supposed to be UEFI with a
> /boot/efi mount point? This seems like a mismatch.

No, it's intended. The UEFI spec supports both msdos and GPT partitioning but quite a lot of the Arm devices we support where we have to write out the firmware at the beginning of the OS disk do not support booting from GPT devices, and in some cases their firmware even gets written through the middle of bits of the GPT partitioning.

Comment 33 Chris Murphy 2020-08-10 22:06:39 UTC
Small issue:

>Metadata,DUP: Size:446.00MiB, Used:162.48MiB (36.43%)
>   /dev/mapper/loop0p3	 892.00MiB
>
>System,DUP: Size:8.00MiB, Used:16.00KiB (0.20%)
>   /dev/mapper/loop0p3	  16.00MiB


What's the most typical use case for the image? SD Card or USB enclosed drive (SSD or HDD)? I wonder if we should make the image 'mkfs.btrfs -msingle'? (a) image size will be smaller (b) 50% the fs metadata writes.

Comment 34 Peter Robinson 2020-08-10 22:34:03 UTC
(In reply to Chris Murphy from comment #33)
> Small issue:
> 
> >Metadata,DUP: Size:446.00MiB, Used:162.48MiB (36.43%)
> >   /dev/mapper/loop0p3	 892.00MiB
> >
> >System,DUP: Size:8.00MiB, Used:16.00KiB (0.20%)
> >   /dev/mapper/loop0p3	  16.00MiB

I'm not sure what I'm looking at.

> What's the most typical use case for the image? SD Card or USB enclosed
> drive (SSD or HDD)? I wonder if we should make the image 'mkfs.btrfs
> -msingle'? (a) image size will be smaller (b) 50% the fs metadata writes.

Both SD card and other drives (Various USB, ATA, NVME, UFS, eMMC drives) are all used.

Comment 35 Chris Murphy 2020-08-10 23:57:22 UTC
(In reply to Peter Robinson from comment #34)
> I'm not sure what I'm looking at.

Sorry, that's a bit of btrfs jargon, that I also trimmed. It's partial output from 'btrfs filesystem usage' and what's it's showing is metadata is using the "DUP" profile. That means there's two copies of the file system on this image.

> Both SD card and other drives (Various USB, ATA, NVME, UFS, eMMC drives) are
> all used.

Almost every flash drive controller out there puts concurrent writes on the same erase block, many also do opportunistic dedup/compress of concurrent writes. Flash failure modes tend not to be random cell failures but rather anything on a given erase block will come back as either zeros or garbage. The duplicate metadata isn't likely to do any good. This applies to both cheap and high end flash.

Comment 36 Chris Murphy 2020-08-11 00:09:42 UTC
While mkfs.btrfs does look at /sys/block/<node>/queue/rotational to decide whether to do DUP or single, (a) virtioblk reports rotational (b) raw images on rotating drives on loop are reported as rotational (c) most all USB sticks report as rotational (d) quite a lot of USB-SATA bridge chipsets report rotational even when an SSD is in the USB enclosure. So it's not reliable.

Plausible the resize tool could also check for rotational. If yes, ask the user a qualifier: is it really a rotating hard drive (not flash media)? If yes, it's simple and safe to convert from metadata single to metadata dup. And now they have some extra chance of self-healing of the file system if there's a problem like a bad sector or torn/misdirected write.

Comment 37 Peter Robinson 2020-08-11 08:29:23 UTC
> Plausible the resize tool could also check for rotational. If yes, ask the
> user a qualifier: is it really a rotating hard drive (not flash media)? If
> yes, it's simple and safe to convert from metadata single to metadata dup.
> And now they have some extra chance of self-healing of the file system if
> there's a problem like a bad sector or torn/misdirected write.

The arm-image-installer resize option runs on a machine that is not the place where the instance will run, I believe the systemd resize tool that is done on a running instance of the OS requires GPT.

Comment 38 Ben Cotton 2020-08-11 13:46:09 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 33 development cycle.
Changing version to 33.

Comment 39 Chris Murphy 2020-08-11 16:27:32 UTC
>The arm-image-installer resize option runs on a machine that is not the place where the instance will run

Oh, then the interrogatory would always have to happen.

>I believe the systemd resize tool that is done on a running instance of the OS requires GPT.

Yeah I'm discounting systemd-repart because of that, and also it's non-interactive. The only way to resolve the ambiguity right now is to ask a human.

But dup metadata is sort of a small issue. A bigger gain would be to (a) set compress=zstd:1 in fstab, and (b) enhance the image builder to use compress-force=zstd:9+ and drop the xz step.

Comment 40 Peter Robinson 2020-08-11 16:46:07 UTC
> But dup metadata is sort of a small issue. A bigger gain would be to (a) set
> compress=zstd:1 in fstab, and (b) enhance the image builder to use
> compress-force=zstd:9+ and drop the xz step.

I feel that's a parallel problem to this bug. Has FESCo approved compression?

Comment 41 Neal Gompa 2020-08-11 16:48:38 UTC
(In reply to Peter Robinson from comment #40)
> > But dup metadata is sort of a small issue. A bigger gain would be to (a) set
> > compress=zstd:1 in fstab, and (b) enhance the image builder to use
> > compress-force=zstd:9+ and drop the xz step.
> 
> I feel that's a parallel problem to this bug. Has FESCo approved compression?

It's part of the approved Change as part of the stretch-goal items.

Comment 42 Chris Murphy 2020-08-11 19:43:57 UTC
Yep, compression is out of scope for this bug. Filed bug 1868142.

Comment 43 Chris Murphy 2020-08-11 19:55:48 UTC
OK I just saw this come in, so it may be that I was confused about these fixes being already in production...
https://pagure.io/fedora-infrastructure/issue/9138#comment-670706

Comment 44 Chris Murphy 2020-08-17 06:09:59 UTC
The mount order seems to be setup here, but then I lose track of what creates and does the assembly. I'm not sure if the confusion happens here, or if the later processes fail to honor the proper ordering.
https://pagure.io/appliance-tools/blob/master/f/appcreate/partitionedfs.py#_310

Comment 45 Neal Gompa 2020-08-23 05:43:02 UTC
I think I've got a solution for this now: https://pagure.io/appliance-tools/pull-request/15

Davide is going to verify this for me, as I'm too tired and my computer is throwing weird errors now with networking in chroots...

Comment 46 Davide Cavalca 2020-08-23 16:56:00 UTC
We think we have a working fix for this now. Neal is cutting a release for appliance-tools, and we'll see how the next compose goes.

Comment 47 Neal Gompa 2020-08-23 17:06:08 UTC
New appliance-tools pushed into F33 which should fix this: https://koji.fedoraproject.org/koji/buildinfo?buildID=1598804

Comment 48 Neal Gompa 2020-08-24 01:44:31 UTC
For some reason, the new Xfce image build did not use my updated appliance-tools build: https://koji.fedoraproject.org/koji/taskinfo?taskID=49954670

Comment 49 Neal Gompa 2020-08-24 03:34:10 UTC
It looks like the Rawhide/F34 Xfce armhfp image build *did* use the updated appliance-tools build and seems to have setup the disk structure correctly: https://koji.fedoraproject.org/koji/taskinfo?taskID=50015418

Comment 50 Neal Gompa 2020-08-24 04:03:51 UTC
Unfortunately, it looks like we need some tweaking here for *unmounting*, based on the the Rawhide image build logs...

DEBUG util.py:621:  umount:
DEBUG util.py:621:  /var/tmp/imgcreate-stkot1rh/install_root/var/cache/dnf:
DEBUG util.py:621:  target
DEBUG util.py:621:  is
DEBUG util.py:621:  busy.
DEBUG util.py:621:  Unable to unmount /var/tmp/imgcreate-stkot1rh/install_root/var/cache/dnf normally, using lazy unmount
DEBUG util.py:621:  lazy umount succeeded on /var/tmp/imgcreate-stkot1rh/install_root/var/cache/dnf
DEBUG util.py:621:  umount: /var/tmp/imgcreate-stkot1rh/install_root/: target is busy.
DEBUG util.py:621:  Unmounting directory /var/tmp/imgcreate-stkot1rh/install_rootbtrfs.007
DEBUG util.py:621:  Unmounting directory /var/tmp/imgcreate-stkot1rh/install_root/boot/efi
DEBUG util.py:621:  Unmounting directory /var/tmp/imgcreate-stkot1rh/install_root/boot
DEBUG util.py:621:  Removing compat symlinks
DEBUG util.py:621:  Unmapping /dev/loop0
DEBUG util.py:621:  Unable to create appliance : Failed to unmap partitions for '/dev/loop0'
DEBUG util.py:621:  umount: /var/tmp/imgcreate-stkot1rh/install_root/home: not mounted.

Comment 51 Neal Gompa 2020-08-24 04:08:31 UTC
Most likely, what's going wrong here is that unmounting is not quite ordered right. It should be: /boot/efi, then /boot, then /home, then /, then the main btrfs volume.

Comment 52 Neal Gompa 2020-08-24 14:32:03 UTC
Part 2 fix, this time for unmounting: https://pagure.io/appliance-tools/pull-request/16

Can someone take this patch and test it?

Comment 53 Neal Gompa 2020-08-24 16:07:09 UTC
Davide confirmed it fixed it for him, so I merged it and released 010.2.

Fedora 33 build: https://koji.fedoraproject.org/koji/buildinfo?buildID=1599804

Fedora Rawhide build: https://koji.fedoraproject.org/koji/buildinfo?buildID=1599801

Comment 54 Paul Whalen 2020-08-24 23:20:42 UTC
Tested appliance-tools-010.2-1.fc32, image created looks good with everything where it should be.

Comment 55 Neal Gompa 2020-08-25 11:53:46 UTC
Last night's Rawhide image build succeeded with everything looking like it was constructed properly: https://koji.fedoraproject.org/koji/taskinfo?taskID=50120252

Comment 56 Paul Whalen 2020-08-25 16:03:52 UTC
Testing the images from Fedora-33-20200825.n.0, the kernel parameters include "root=None" which causes an immediate boot failure. Updating that with the expected root=UUID=* it boots further (then hits other unrelated issues).

Comment 57 Davide Cavalca 2020-08-25 16:15:30 UTC
Thanks Paul! I suspect that's a bug in _get_grub_boot_config in appliance.py, as I think that's what feeds this parameter.

Comment 58 Paul Whalen 2020-08-25 16:38:36 UTC
Looking further, root get mounted RO, which causes a bunch of services to fail. Remounting RW and restarting the failed services worked.

Comment 59 Neal Gompa 2020-08-26 00:12:36 UTC
So here's another attempt to get this working: https://pagure.io/appliance-tools/pull-request/17

Comment 60 Fedora Update System 2020-08-27 03:19:50 UTC
FEDORA-2020-c6d9203505 has been submitted as an update to Fedora 33. https://bodhi.fedoraproject.org/updates/FEDORA-2020-c6d9203505

Comment 61 Fedora Update System 2020-08-27 03:19:51 UTC
FEDORA-2020-9278467e30 has been submitted as an update to Fedora 31. https://bodhi.fedoraproject.org/updates/FEDORA-2020-9278467e30

Comment 62 Fedora Update System 2020-08-27 03:19:52 UTC
FEDORA-2020-97563020d8 has been submitted as an update to Fedora 32. https://bodhi.fedoraproject.org/updates/FEDORA-2020-97563020d8

Comment 63 Fedora Blocker Bugs Application 2020-08-27 03:21:02 UTC
Proposed as a Blocker for 33-beta by Fedora user ngompa using the blocker tracking app because:

 We cannot build working 32-bit ARM based images for desktop variants without this fix.

Comment 64 Fedora Update System 2020-08-27 14:21:15 UTC
FEDORA-2020-9278467e30 has been pushed to the Fedora 31 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 65 Fedora Update System 2020-08-27 14:31:20 UTC
FEDORA-2020-97563020d8 has been pushed to the Fedora 32 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 66 Fedora Update System 2020-08-27 14:46:59 UTC
FEDORA-EPEL-2020-cf6e415f7a has been pushed to the Fedora EPEL 8 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 67 Fedora Update System 2020-08-27 19:42:24 UTC
FEDORA-2020-c6d9203505 has been submitted as an update to Fedora 33. https://bodhi.fedoraproject.org/updates/FEDORA-2020-c6d9203505

Comment 68 Adam Williamson 2020-08-27 23:19:33 UTC
There's +4 for BetaFE in the vote ticket - https://pagure.io/fedora-qa/blocker-review/issue/44 - so marking accepted FE. I'm gonna just go ahead and unpropose it as a Beta blocker as it clearly can't be one, as Chris pointed out in the vote ticket, no release blocking image is affected by the bug.

Comment 69 Adam Williamson 2020-08-27 23:20:19 UTC
Re-opening, as this is reported against 33. 31 update should not have been marked as fixing it.

Comment 70 Fedora Update System 2020-08-28 02:38:03 UTC
FEDORA-2020-c6d9203505 has been pushed to the Fedora 33 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 71 Neal Gompa 2020-08-28 11:58:56 UTC
Confirmed everything is fixed now in https://koji.fedoraproject.org/koji/taskinfo?taskID=50314607

ngompa@f32-ryo-ohki-winvm ~/t/a/F/etc> cat extlinux.conf
# extlinux.conf generated by appliance-creator
ui menu.c32
menu autoboot Welcome to Fedora-Xfce-armhfp-33-20200828.n.0. Automatic boot in # second{,s}. Press a key for options.
menu title Fedora-Xfce-armhfp-33-20200828.n.0 Boot Options.
menu hidden
timeout 20
totaltimeout 600

label Fedora-Xfce-armhfp-33-20200828.n.0 (5.8.3-300.fc33.armv7hl)
	kernel /vmlinuz-5.8.3-300.fc33.armv7hl
	append ro root=UUID=f755ea1a-e7f3-49ab-b9fa-9e4a37ad4989 rhgb quiet LANG=en_US.UTF-8 rootflags=subvol=root cma=192MB
	fdtdir /dtb-5.8.3-300.fc33.armv7hl/
	initrd /initramfs-5.8.3-300.fc33.armv7hl.img

ngompa@f32-ryo-ohki-winvm ~/t/a/F/etc> cat fstab
UUID=ea69aa9d-8239-4b20-b9ce-ae8c00c9d332  /boot ext4    defaults,noatime 0 0
UUID=5884-1077  /boot/efi vfat    defaults,noatime 0 0
UUID=f755ea1a-e7f3-49ab-b9fa-9e4a37ad4989  /home btrfs    subvol=home,noatime 0 0
UUID=f755ea1a-e7f3-49ab-b9fa-9e4a37ad4989  / btrfs    subvol=root,noatime 0 0