Bug 2073708

Created attachment 1871611 [details]
File: anaconda-tb

Created attachment 1871612 [details]
File: anaconda.log

Created attachment 1871613 [details]
File: dbus.log

Created attachment 1871614 [details]
File: environ

Created attachment 1871615 [details]
File: journalctl

Created attachment 1871616 [details]
File: lsblk_output

Created attachment 1871617 [details]
File: lvm.log

Created attachment 1871618 [details]
File: nmcli_dev_list

Created attachment 1871619 [details]
File: os_info

Created attachment 1871620 [details]
File: program.log

Created attachment 1871621 [details]
File: storage.log

It seems anaconda mounts /dev/rhel/home right when "configuring storage" phase starts and crashes hard on this (tried umount /dev/rhel/home, started anaconda again, configured storage, checked lsblk, /dev/rhel/home wasn't mounted, started installation, which crashed with /dev/rhel/home mounted).

The content on /dev/sda is some el9 compose from late January of this year.

Similar problem has been detected:

Was trying to install F36 over an old Fedora install. Installed couple years back as F27(?), currently F33.

I used the "Custom" option for partitioning because I wanted to keep the home partition if possible. / and /home are in an LVM group that was set up by the F27 install way back.

After assigning all the necessary mountpoints, I proceeded, couple seconds later I got the error message "device is active". The first time around I noticed the partition was mounted, so I unmonted it manually thinking that's what was causing the issue.
A second attempt showed me this was not the case, as the installer seems to mount the partition on its own (and presumably on purpose).

Trying a 3rd time I noticed I cannot configure or reformat the swap partition and /boot mountpoint anymore, all the fields were greyed out (including the "Reformat" checkbox).
This log is from an attempt at using the Blivet-GUI option instead which let me make all the required changes, but still failed to do the installation.


addons:         com_redhat_kdump
cmdline:        /usr/bin/python3  /sbin/anaconda --liveinst --graphical
cmdline_file:   BOOT_IMAGE=/images/pxeboot/vmlinuz root=live:CDLABEL=Fedora-KDE-Live-36_B-1-4 rd.live.image quiet rhgb
dnf.librepo.log: 
hashmarkername: anaconda
kernel:         5.17.0-0.rc7.116.fc36.x86_64
other involved packages: python3-libs-3.10.2-3.fc36.x86_64, python3-dasbus-1.6-4.fc36.noarch
package:        anaconda-core-36.16.2-4.fc36.x86_64
packaging.log:  
product:        Fedora
reason:         pyanaconda.modules.common.errors.installation.StorageInstallationError: device is active
release:        Fedora release 36 (Thirty Six)
release_type:   pre-release
version:        36

So, the reproducer I've came down to is:

1. Boot up RHEL 9 (tested on RHEL-9.0.0-20220410.d.0-x86_64-dvd1.iso )
2. In storage configuration, choose custom
3. Use lvm in custom, add separate /home
4. Finish installation
5. Boot up Fedora 36 KDE Live (this doesn't happen on Workstation)
6. Start installation, automatic storage configuration
7. RHEL's /home gets mounted, "device is active" appears, which is followed by this crash when dismissed

Proposing for Blocker discussion.

I should mention for me the installer didn't "crash", i.e. it didn't force-close. But I had no choice then to click Quit myself (no other option there anyway).

As a workaround I used KDE Partition manager to delete the volume manually. Then in the installer I used Blivet-GUI to create a new volume and mount point on it. The installer ran through flawlessly and I'm gonna assume the installation will work (if not I'll comment again).

DEBUG:blivet:                XFS.destroy: device: /dev/mapper/rhel-home ; type: xfs ; status: True ;
ERROR:anaconda.modules.storage.installation:Failed to create storage layout: device is active
Traceback (most recent call last):
  File "/usr/lib64/python3.10/site-packages/pyanaconda/modules/storage/installation.py", line 82, in run
    self._turn_on_filesystems(
  File "/usr/lib64/python3.10/site-packages/pyanaconda/modules/storage/installation.py", line 162, in _turn_on_filesystems
    storage.do_it(callbacks)
  File "/usr/lib/python3.10/site-packages/blivet/threads.py", line 53, in run_with_lock
    return m(*args, **kwargs)
  File "/usr/lib/python3.10/site-packages/blivet/blivet.py", line 114, in do_it
    self.devicetree.actions.process(callbacks=callbacks, devices=self.devices)
  File "/usr/lib/python3.10/site-packages/blivet/actionlist.py", line 48, in wrapped_func
    return func(obj, *args, **kwargs)
  File "/usr/lib/python3.10/site-packages/blivet/actionlist.py", line 328, in process
    action.execute(callbacks)
  File "/usr/lib/python3.10/site-packages/blivet/threads.py", line 53, in run_with_lock
    return m(*args, **kwargs)
  File "/usr/lib/python3.10/site-packages/blivet/deviceaction.py", line 760, in execute
    self.format.destroy()
  File "/usr/lib/python3.10/site-packages/blivet/threads.py", line 53, in run_with_lock
    return m(*args, **kwargs)
  File "/usr/lib/python3.10/site-packages/blivet/formats/__init__.py", line 552, in destroy
    self._pre_destroy(**kwargs)
  File "/usr/lib/python3.10/site-packages/blivet/threads.py", line 53, in run_with_lock
    return m(*args, **kwargs)
  File "/usr/lib/python3.10/site-packages/blivet/formats/__init__.py", line 562, in _pre_destroy
    raise DeviceFormatError("device is active")
blivet.errors.DeviceFormatError: device is active
INFO:anaconda.threading:Thread Failed: AnaTaskThread-CreateStorageLayoutTask-1 (140504041707072)
ERROR:anaconda.modules.common.task.task:Thread AnaTaskThread-CreateStorageLayoutTask-1 has failed: Traceback (most recent call last):
  File "/usr/lib64/python3.10/site-packages/pyanaconda/threading.py", line 275, in run
    threading.Thread.run(self)
  File "/usr/lib64/python3.10/threading.py", line 946, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib64/python3.10/site-packages/pyanaconda/modules/common/task/task.py", line 96, in _thread_run_callback
    self._task_run_callback()
  File "/usr/lib64/python3.10/site-packages/pyanaconda/modules/common/task/task.py", line 109, in _task_run_callback
    self._set_result(self.run())
  File "/usr/lib64/python3.10/site-packages/pyanaconda/modules/storage/installation.py", line 94, in run
    raise StorageInstallationError(str(e)) from None
pyanaconda.modules.common.errors.installation.StorageInstallationError: device is active

Reassigning to blivet for further investigation.

Discussed during the 2022-04-11 blocker review meeting: [1]

The decision to classify this bug as an AcceptedBlocker was made:

"It violates the following criterion: “When using the guided partitioning flow, the installer must be able to… Cleanly install to a disk with a valid ms-dos or gpt disk label and partition table which contains existing data and sufficient unpartitioned space for a Fedora installation"."

[1] https://meetbot-raw.fedoraproject.org/fedora-blocker-review/2022-04-11/f36-blocker-review.2022-04-11-16.00.log.txt

The /home logical volume gets mounted by UDisks during the storage configuration:

Apr 09 23:05:40 localhost-live udisksd[1269]: Mounted /dev/dm-2 at /run/media/liveuser/1f3c9b06-1e30-485c-8cb4-af22bb7b28b7 on behalf of uid 1000

unfortunately this happens right before we try to remove the LV and Blivet is not able to react to changes from the outside. The mount happens directly after we activate the LV (we activate logical volumes before removing to be able to wipe the file system from them) so something in KDE reacts to the activation (and sudden appearance of a new block device) and decides to mount it. I'm not sure what part of KDE is responsible for this (maybe kf5-kio?), UDisks itself doesn't automount devices so something from KDE told UDisks to mount it.

Created attachment 1872418 [details]
KDE automount settings

I am not able to reproduce the issue if I disable "Removable Devices" -> "All Known Devices" -> "On Attach:.

Reassigning to spin-kickstarts.

This seems more like a bug in KDE deciding what's a "removable device" than any kind of downstream config issue, to me. We *want* the live image to automount "removable devices", but unless the testers are using external USB disks to test with or something, these devices should not be considered "removable" and automounted by KDE...

Re-assigning to a general KDE-y component, because I never know which bit of KDE is responsible for what. This should at least get the right people to see it.

Frantisek, can you see if this happens with F35 KDE?

This happen on F35 KDE too, tested today.

Created attachment 1874211 [details]
ext4 and LVM setup on VM for test with F35

Created attachment 1874212 [details]
anaconda setup at F35 install

Created attachment 1874213 [details]
error at anaconda F35 KDE install

Created attachment 1874214 [details]
same outputs at F35 as at F36

For reference, I used f35 kde respin iso, from 2022/04/15

So I played with this some myself, and it's interesting. I reproduced it once on F35 and once on F36, but then on some further tries, it worked OK. On F36 I see some kind of KDE dialog pop up very briefly - when it failed, the error appeared right after that; when it works, that dialog just goes away and the install process continues. So I think this may also be a bit racy?

I did not mange to reproduce this on F34, but F34's automount settings look exactly the same as F35's, so I suspect I just didn't "win" the race on either try with F34. F36's automount settings look slightly different - there's this "All Known Devices" setting checked by default which isn't really there in F34/F35.

I'll take a few more shots with 34 and 35...

After trying this a few more times, I still can't reproduce it again on F34 or F35. I'm almost not sure it really happened any more. The bug happens about half the time for me on F36.

It does seem like something changed here in F36. I think it's, more or less, this commit upstream:

https://github.com/KDE/plasma-desktop/commit/f27e6618f47e2df44aa7c6f53eff3732896dc88a

Before that - in F34 and F35 - there's a kinda overall checkbox called "Enable automatic mounting of removable media", which is unchecked by default. The help text explains that no automatic mounting will happen if it's unchecked, and the other settings underneath it are greyed out to indicate this.

In F36, the overall checkbox is renamed "Automatically mount removable media that have never been mounted before", and is unchecked by default. But now this is clearly referring only to 'unknown' devices, and it's at the bottom of the window. There is an active "All Known Devices" setting at the top of the window that's checked on for "On Login" and "On Attach".

This kinda looks a lot like F34 and F35 were set not to automount anything, but F36 is set to automount "known devices" on attach.

There's still the issue that this whole pane is titled "Removable Devices", but we're clearly dealing with devices that are not 'removable' getting automounted.

I tried now with the original release F35 kde iso (Fedora-KDE-Live-x86_64-35-1.2.iso) twice, and anaconda run without issues, no bug. Installation completed just fine.
Will try again with the respin iso to check it out.

Yup, tested twice with F35-KDE-x86_64-LIVE-20220415.iso (respin iso) and the bug is there.
Tested twice with oficial release iso (Fedora-KDE-Live-x86_64-35-1.2.iso) and no bug.

It looks like plasma-desktop has been updated to 5.24.4 in F35 and that version has the same changes I mentioned in #c30, so that makes sense in fact.

I'm now testing a change that would disable the "On Login" and "On Attach" boxes for "All Known Devices" - both to see if it fixes the bug, and how it affects what happens when you plug in a USB stick, and how that compares to F35.

So I think I've cracked this. https://pagure.io/fedora-kickstarts/pull-request/889 should fix it.

That PR writes a config file to the live environment so that automount of "known devices" is disabled. I built a live image with this change included:

https://openqa.stg.fedoraproject.org/tests/1719029/asset/iso/01719029-Fedora-KDE-Live-x86_64-FEDORA-2022-77f3c1cd62.iso

and tested it five times, and did not hit the bug once (it usually shows up about one in every two tries).

I also tested how it affects actual automount behaviour, compared to Fedora 35 (original release). In all cases - F35, F36, and F36 with the change - when you first plug in a USB stick, it is not mounted. KDE shows a notification, which has a button to mount the stick. If you mount the stick, then unmount it and unplug it, the behaviour differs when you plug it back in. On Fedora 35, when you plug it back in, nothing happens - the notification does not appear again, and the stick is not automounted. You have to go to a file manager or open the USB devices notification panel yourself to mount it. On Fedora 36 without this change, the second time you plug the stick in, it gets automounted - which I think is the same behaviour that's affecting anaconda here. On Fedora 36 with my change, the second time you plug the stick in, it is not automounted, but the notification is shown again (which seems like an improvement on F35).

So, I think this is good. It seems to solve the problem and does not make the behaviour with actual USB sticks any worse than F35 was. The automount-on-second-plugin thing is I guess very slightly handy, but no big deal, especially in a live environment.

It's nice to see some activity here and that you found a way around this issue.
However I feel like this still leaves the question of why an internal hard drive is considered "removable" in the first place. Should this not be fixed upstream instead of patchworking around it?

Also, how does it affect the setting after installation?

Tarulia: yes, I do consider that kind of an issue. I did poke down that avenue a little bit, but I didn't find any nice easy points of entry to try and figure out what bit of KDE decides when something is "removable" and whether we can convince it that hard disks aren't. (Even if we did that, there are still corner cases; you can install Fedora or RHEL on a USB-attached external disk, for instance. Should that be considered a "removable" device that should be automounted?) This was easier, and it sufficiently deals with the blocker that we need to fix on a short timeline, so I did this instead.

It does not affect the setting after installation because the change is only applied to the live environment. On the installed system automount of "known devices" will be enabled.

Neal merged the PR, so we can call this ON_QA now.

Adam I tested your build ( https://openqa.stg.fedoraproject.org/tests/1719029/asset/iso/01719029-Fedora-KDE-Live-x86_64-FEDORA-2022-77f3c1cd62.iso ) here, twice, and it worked fine, no bugs. Congrats :)

Can folks confirm the fix in Final RC1 too? Thanks!

(In reply to Adam Williamson from comment #39)
> Can folks confirm the fix in Final RC1 too? Thanks!

Tested with Fedora-KDE-Live-x86_64-36-1.1.iso and it works fine here :D

Thanks!

Tested too with Fedora-Workstation-Live-x86_64-36-1.1.iso and bug is fixed too. So I think we are good to go on that.

I guess we can actually call this closed now since the fix was in a kickstart and is merged already.