Bug 1348688 - Anaconda cannot access LVM partitions in a LUKS-encrypted disk partition after decryption
Summary: Anaconda cannot access LVM partitions in a LUKS-encrypted disk partition afte...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: LVM and device-mapper
Classification: Community
Component: lvm2
Version: 2.02.150
Hardware: x86_64
OS: Other
unspecified
unspecified
Target Milestone: Fedora
: ---
Assignee: Peter Rajnoha
QA Contact: cluster-qe@redhat.com
URL:
Whiteboard: https://fedoraproject.org/wiki/Common...
: 1350947 (view as bug list)
Depends On:
Blocks: F26BetaBlocker 1351883
TreeView+ depends on / blocked
 
Reported: 2016-06-21 19:07 UTC by Steve Bryant
Modified: 2017-06-03 17:38 UTC (History)
27 users (show)

Fixed In Version: lvm2-2.02.168-6.fc26
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1351883 (view as bug list)
Environment:
Last Closed: 2017-06-03 17:38:55 UTC
rule-engine: lvm-technical-solution?
rule-engine: lvm-test-coverage?


Attachments (Terms of Use)
anaconda.log (26.54 KB, text/plain)
2016-06-21 22:15 UTC, Steve Bryant
no flags Details
dnf.librepo.log (124.08 KB, text/plain)
2016-06-21 22:16 UTC, Steve Bryant
no flags Details
dnf.log (2.22 KB, text/plain)
2016-06-21 22:16 UTC, Steve Bryant
no flags Details
dnf.rpm.log (49 bytes, text/plain)
2016-06-21 22:17 UTC, Steve Bryant
no flags Details
hawkey.log (1.96 KB, text/plain)
2016-06-21 22:17 UTC, Steve Bryant
no flags Details
ifcfg.log (2.42 KB, text/plain)
2016-06-21 22:18 UTC, Steve Bryant
no flags Details
packaging.log (699 bytes, text/plain)
2016-06-21 22:19 UTC, Steve Bryant
no flags Details
program.log (40.69 KB, text/plain)
2016-06-21 22:19 UTC, Steve Bryant
no flags Details
sensitive-info.log (106 bytes, text/plain)
2016-06-21 22:20 UTC, Steve Bryant
no flags Details
storage.log (161.57 KB, text/plain)
2016-06-21 22:20 UTC, Steve Bryant
no flags Details
syslog (82.37 KB, text/plain)
2016-06-23 16:53 UTC, Steve Bryant
no flags Details
X.log (27.53 KB, text/plain)
2016-06-23 16:54 UTC, Steve Bryant
no flags Details
updates (10.15 KB, application/octet-stream)
2017-05-19 09:53 UTC, Marian Csontos
no flags Details

Description Steve Bryant 2016-06-21 19:07:33 UTC
Description of problem:
On accessing the MANUAL PARTITIONING screen, my hard disk layout is displayed as follows

 Unknown
  ext4
  sda1
  Encrypted (LUKS)
  sda2
  BIOS boot
  sda9

On selecting the "Encrypted (LUKS)" partition, I can enter my passphrase and decrypt the device.

Anaconda now shows my hard disk layout as follows

 Unknown
  physical volume (LVM)
  luks-123....
  ext4
  sda1
  BIOS boot
  sda9

I can find no way to access and use my LVM configuration on the LVM physical volume.  However, from the shell I can see the partition has been decrypted and Volume Group and all the Logical Volumes contained have been activated.

It seems like in Fedora 24 Anaconda simply does not try to detect any LVM configuration within a LUKS-encrypted partition when it is decrypted.

Version-Release number of selected component (if applicable):
(as shipped in Fedora-Workstation-netinst-x86_64-24-1.2.iso)

How reproducible:


Steps to Reproduce:
1. LUKS-encrypt a disk partition
2. Decrypt the partition and create LVM Volume Group using the partion.
3. Create some Logical Volumes in the VG
4. Reboot using Fedora 24 installation media and try to access the LVs when creating a manual partition layout. 

Actual results:
After decrypting the LUKS partition, Anaconda only recognizes the contents as an LVM Physical Volume and so the only option available is to reformat.

Expected results:
Anaconda detects the LVM configuration on the Physical Volume, and allows the LVs to be added to the filesystem tree.

Additional info:
I was able to use this disk layout when installing Fedora 23 (admittedly I have converted the hdd format from MBR to GDM since then).

Comment 1 Brian Lane 2016-06-21 21:53:19 UTC
Please attach the logs from /tmp/*log as individual text/plain attachments.

Comment 2 Steve Bryant 2016-06-21 22:15:22 UTC
Created attachment 1170431 [details]
anaconda.log

Comment 3 Steve Bryant 2016-06-21 22:16:25 UTC
Created attachment 1170432 [details]
dnf.librepo.log

Comment 4 Steve Bryant 2016-06-21 22:16:51 UTC
Created attachment 1170433 [details]
dnf.log

Comment 5 Steve Bryant 2016-06-21 22:17:27 UTC
Created attachment 1170434 [details]
dnf.rpm.log

Comment 6 Steve Bryant 2016-06-21 22:17:59 UTC
Created attachment 1170435 [details]
hawkey.log

Comment 7 Steve Bryant 2016-06-21 22:18:29 UTC
Created attachment 1170436 [details]
ifcfg.log

Comment 8 Steve Bryant 2016-06-21 22:19:02 UTC
Created attachment 1170437 [details]
packaging.log

Comment 9 Steve Bryant 2016-06-21 22:19:34 UTC
Created attachment 1170438 [details]
program.log

Comment 10 Steve Bryant 2016-06-21 22:20:10 UTC
Created attachment 1170439 [details]
sensitive-info.log

Comment 11 Steve Bryant 2016-06-21 22:20:46 UTC
Created attachment 1170440 [details]
storage.log

Comment 12 Steve Bryant 2016-06-21 22:27:48 UTC
Cannot attach "syslog" and "X.log" - perhaps 10-attachment limit?

If you require either of these, can you nominate a file to be replaced?

Comment 13 Steve Bryant 2016-06-21 23:12:50 UTC
Note that the LUKS partition is the only PV used by VG "vg_miles":

# pvs
  PV                                                    VG       Fmt  Attr PSize   PFree  
  /dev/mapper/luks-5b484048-e56e-4c04-be51-bba5553a3752 vg_miles lvm2 a--  464.75g 280.75g

Comment 14 Rob Foehl 2016-06-23 01:32:57 UTC
Same problem here, LVM volumes within LUKS aren't recognized by Anaconda after successfully unlocking the LUKS volume.  Correct entries exist in /dev/mapper, pvs output is correct, ... but Anaconda never notices.  Affects the netinst media for both Workstation and Server, haven't tried anything else thus far.

Comment 15 Steve Bryant 2016-06-23 07:39:21 UTC
I think I might see the issue - in "/tmp/storage.log" (attached), it actually finds the devices used by LVM volumes listed in /dev/mapper:

> 23:01:40,132 DEBUG storage.ui:            DeviceTree.getDeviceByName: hidden: False ; incomplete: False ; name: vg_miles-libvirt ;
> 23:01:40,133 DEBUG storage.ui:            DeviceTree.getDeviceByName returned None
> 23:01:40,134 INFO storage.ui: vg_miles-libvirt is an lvm logical volume

However, it fails to find a corresponding device for the LVM volume group:

> 23:01:40,150 DEBUG storage.ui:             DeviceTree.getDeviceByName: hidden: False ; incomplete: False ; name: vg_miles ;
> 23:01:40,152 DEBUG storage.ui:             DeviceTree.getDeviceByName returned None
> 23:01:40,152 ERR storage.ui: failed to find vg 'vg_miles' after scanning pvs

This is to be expected as LVM volume groups have no corresponding device special file ("DeviceTree.getDeviceByName" will always fail for a VG name?)  In the "/dev" file system LVM volume groups only exist as directories containing links from the names of the logical volumes they contain to the corresponding "dm" devices:

# find /dev -name vg_miles
/dev/vg_miles

# ls -lh /dev/vg_miles
total 0
lrwxrwxrwx. 1 root root 7 Jun 23 07:40 F21Root -> ../dm-5
lrwxrwxrwx. 1 root root 7 Jun 23 07:40 F23Root -> ../dm-1
lrwxrwxrwx. 1 root root 7 Jun 23 07:40 F24Root -> ../dm-7
lrwxrwxrwx. 1 root root 7 Jun 23 07:40 Home -> ../dm-6
lrwxrwxrwx. 1 root root 7 Jun 23 07:40 lfs -> ../dm-3
lrwxrwxrwx. 1 root root 7 Jun 23 07:40 libvirt -> ../dm-4
lrwxrwxrwx. 1 root root 7 Jun 23 07:40 Swap -> ../dm-2

Comment 16 David Lehman 2016-06-23 16:37:45 UTC
It looks like we can't get information about the recently-decrypted PV. Now that we're using the lvm-dbus plugin I can't see what the PV info looks like -- maybe the path for a device-mapper PV isn't using the map name (using 'dm-2' instead?).

Comment 17 Steve Bryant 2016-06-23 16:53:36 UTC
Created attachment 1171614 [details]
syslog

File permissions were preventing me uploading this...

Comment 18 Steve Bryant 2016-06-23 16:54:50 UTC
Created attachment 1171615 [details]
X.log

Comment 19 Vratislav Podzimek 2016-06-24 08:49:33 UTC
(In reply to David Lehman from comment #16)
> It looks like we can't get information about the recently-decrypted PV. Now
> that we're using the lvm-dbus plugin I can't see what the PV info looks like
> -- maybe the path for a device-mapper PV isn't using the map name (using
> 'dm-2' instead?).

The problem here is that the lvmdbusd doesn't notice and react on the even of the LUKS device being opened. So while the 'pvs' utility knows about the PV, which means that the pvscan was performed, the DBus API doesn't know about it. Reassigning this to LVM.

Comment 20 Vratislav Podzimek 2016-06-24 12:36:50 UTC
The following updates.img should work around this issue by avoiding the use of the DBus API: http://vpodzime.fedorapeople.org/1348688_updates.img

Comment 21 Tomas Dolezal 2016-06-24 12:56:12 UTC
I've succeeded in reusing encrypted partitions after specifying this updates= image file.

Comment 22 Steve Bryant 2016-06-24 15:31:28 UTC
The 1348688_updates.img patch also allowed me to install F24 using my LVM configuration on the LUKS-encrypted volume after decryption.

Thanks!

Comment 23 Tony Asleson 2016-06-27 17:06:31 UTC
lvm fedora package isn't being built with '--enable-notify-dbus', thus the lvm dbus service is not getting notifications from lvm to refresh its state.  So at the moment the lvm dbus service is quite broken.  One potential workaround is to change the service file to add back in the '--udev' option until a new package with '--enable-notify-dbus' is available.

Comment 24 Tony Asleson 2016-06-27 22:44:42 UTC
I found a bug when running with 'enable-notify-dbus'.  When doing a lv move, vg move or lv snapshot merge the client will hang if they specify that they willing to wait definitely. 

Immediate work around is to use --udev option when running service instead of compiling with notify dbus support.  Once we get a fix in for this hang then we should compile with 'enable-notify-dbus'.

Comment 25 Tony Asleson 2016-06-28 19:31:58 UTC
Hang when using 'enable-notify-dbus' has been committed upstream.  To reiterate, the issue which pertains to the bz can be addressed by 2 different ways:

1. Repackage existing version, but patch the lvmdbusd service file to use --udev
2. Release a new version of lvm which contains https://git.fedorahosted.org/cgit/lvm2.git/commit/?id=dd5d865020acd545712d4bcc0f3236143de4d76d and utilize 'enable-notify-dbus' when building lvm.

Comment 26 Brian Lane 2016-06-28 21:26:38 UTC
*** Bug 1350947 has been marked as a duplicate of this bug. ***

Comment 27 Peter Rajnoha 2016-07-01 06:41:03 UTC
Fedora 24 anaconda installer is fixed in current version using update image from https://fedoraproject.org/wiki/Common_F24_bugs#lvm-on-luks-reuse-broken. I'll also provide updated lvm2 package in Fedora 24 as well.

Comment 28 Kamil Páral 2016-08-26 08:42:49 UTC
Should this be fixed in F25, does anyone know?

Comment 29 Šimon Lukašík 2016-08-26 19:00:00 UTC
Yesterday in a pub, I heard this has been fixed shortly after 24 GA compose. 

Drunken people are oftentimes honest, so I tend to believe things can get better for Fedora 25, assuming the scotch keeps coming.

Comment 30 Germano Massullo 2016-08-26 19:19:06 UTC
(In reply to Šimon Lukašík from comment #29)
> Yesterday in a pub, I heard this has been fixed shortly after 24 GA compose. 
> 
> Drunken people are oftentimes honest, so I tend to believe things can get
> better for Fedora 25, assuming the scotch keeps coming.

LOL
http://giphy.com/gifs/maury-the-maury-show-l3V0wkQ2KKcAeW8Cs

Comment 31 Vratislav Podzimek 2016-08-29 07:37:34 UTC
(In reply to Šimon Lukašík from comment #29)
> Yesterday in a pub, I heard this has been fixed shortly after 24 GA compose.
I can confirm this...should be fixed. :)

> 
> Drunken people are oftentimes honest, so I tend to believe things can get
> better for Fedora 25, assuming the scotch keeps coming.
And this I can confirm too. :)

Comment 32 Kyle Marek 2016-11-20 00:56:40 UTC
Since my search skills failed me in finding this report and the updates file, my workaround was to do something dangerous and use the Fedora 25 pre-release netinst, with the 24 repo. I came here to document the LVM issue, but then found this report.

Anyway, in addition to that updates image, it appears to work just fine in the Fedora 25 pre-release. I just thought I'd point that out. Hopefully it will continue to work when Fedora 25 is released.

Comment 33 pstodulk 2017-04-29 15:30:41 UTC
Just want to inform you that this bug appears in F26. Could you look at it and fix it? I guess it was build from 27. 04.

Comment 34 pstodulk 2017-04-29 16:12:13 UTC
Just additional info: This is whole process how to use already created partitions on F25 installer:

1) select disk + custom partitioning
2) rescan disk (automatically go back to disk selection...)
3) go again to partitioning
   - now you see unkown parititons
4) unlock luks partition
5) rescan disk again (again you are switched to disk selection ....)
6) and again, go back to partitioning

-- now you can finally continue and select your paritions.. But this works only on F25. For F26 you stuck on step 5.

Is it really necessary? Is there some specific purpose for this pain? Why this is not merged only to two steps (1, 4)? Or at least 1,2,4 (in case that "rescan" could take a lot of time and you don't want to reuse any partition)?

If you really want to keep the process so unfriendly, there would be at least info message informing user they have to rescan disk again and again manually.

Comment 35 pstodulk 2017-04-29 16:17:44 UTC
Heh. I tried that again on F25 and behaviour is different again. So now I didn't have to do steps 5,6. So there are probably some another troubles, when sometimes it requires another rescan and sometimes it doesn't.

Comment 36 pstodulk 2017-04-30 15:12:09 UTC
So, it is because of different behaviour between use of the Unlock button and hit the Enter key. See https://bugzilla.redhat.com/show_bug.cgi?id=1446924

Comment 37 Kamil Páral 2017-05-05 11:28:43 UTC
I haven't tried myself, but based on comment 33 proposing as a beta blocker:
https://fedoraproject.org/wiki/Fedora_26_Beta_Release_Criteria#Custom_partitioning

Comment 38 Adam Williamson 2017-05-08 17:22:12 UTC
Can someone provide a clear explanation of how to reproduce this, starting from scratch (i.e. a VM with an empty hard disk)?

I tried to reproduce this by doing an F25 install using guided partitioning with "encrypt my data" selected, then booting the most recent F26 nightly and going to custom partitioning. It showed the encrypted device under Other, and when I selected it and entered the passphrase, it worked correctly - custom now shows a 'Fedora Linux 25 for x86_64' entry with the relevant LVs in it.

So does this involve a different way of setting up the original install?

Comment 39 Kyle Marek 2017-05-08 20:51:21 UTC
I have reproduced in a VM on a Fedora 25 host with qemu-system-x86 (other non @^minimal-environment packages needed for this process include wget, gdisk, qemu-img, and a desktop environment unless you use SPICE or VNC to view QEMU).

#
# I used this installer
#
wget 'https://mirrors.kernel.org/fedora/development/26/Everything/x86_64/iso/Fedora-Everything-netinst-x86_64-26-20170506.n.0.iso'

#
# Pre-partition an empty disk
# (using host via qemu-nbd)
#

# Create disk image
qemu-img create -f qed fedora.bug.1348688.test.qed 32G

# Open disk image on host
sudo modprobe nbd max_part=64
sudo qemu-nbd -c /dev/nbd0 -f qed fedora.bug.1348688.test.qed

# Partition disk
sudo sgdisk /dev/nbd0 \
  -Z \
  -n=1::+1M -t=1:ef02 -c=1:'BIOS boot partition' \
  -n=2::+64M -t=2:ef00 -c=2:'EFI System Partition' \
  -n=3::+512M -t=3:8300 -c=3:'boot' \
  -n=4:: -t=4:8300 -c=4:'' \
  -p

# Create LUKS
sudo cryptsetup luksFormat /dev/nbd0p4
sudo cryptsetup open --type luks /dev/nbd0p4 cryptroot

# Create VG (named after its own UUID)
sudo lvm pvcreate /dev/mapper/cryptroot
sudo lvm vgcreate vg00 /dev/mapper/cryptroot
eval "$(sudo lvm vgs --nameprefixes --no-headings -o uuid vg00)"
sudo lvm vgrename vg00 "vg_$LVM2_VG_UUID"

# Create root LV
sudo lvm lvcreate -l 100%FREE -n root "vg_$LVM2_VG_UUID"

# Close everything
sudo lvm vgchange -an "vg_$LVM2_VG_UUID"
sudo cryptsetup close cryptroot
sudo qemu-nbd -d /dev/nbd0

#
# Start VM with just-partitioned disk
#
qemu-system-x86_64 \
  -nodefaults -nodefconfig -no-user-config \
  -machine q35,accel=kvm -cpu host -smp cores=$(nproc) -m 2G \
  -drive file="/usr/share/edk2/ovmf/OVMF_CODE.fd",format=raw,if=pflash,readonly=on \
  -device virtio-vga \
  -device virtio-net-pci,netdev=eth0 -netdev user,id=eth0 \
  -drive file="Fedora-Everything-netinst-x86_64-26-20170506.n.0.iso",format=raw,media=cdrom,if=none,id=cdrom0 \
  -device virtio-scsi-pci,id=scsi -device scsi-disk,drive=cdrom0,bus=scsi.0 \
  -drive file="fedora.bug.1348688.test.qed",format=qed,if=virtio \
  -monitor stdio


In installer:

 - Accept fate
 - Select Installation Destination
   - Storage Configuration: Custom
   - Done (Installation Destination)
      - Expand "Unknown" category
      - Select "Encryoted (LUKS)", unlock

Notice vg_whatever-root is not an option (was immediately an option in F25).


Woraround:

 - Main menu with Done (Manual Partitioning) twice
    - Select Installation Destination
       - Click "Refresh..." in bottom right
          - Rescan Disks
       - Done (Installation Destination)

Notice vg_whatever-root is now available.

Comment 40 Mike Ruckman 2017-05-12 21:20:59 UTC
Discussed in 2017-05-08 Blocker review meeting. This bug is a violation of the Beta criterion: "When using the custom partitioning flow, the installer must be able to: ... Correctly interpret, and modify as described below, any disk with a valid ms-dos or gpt disk label and partition table..."

Comment 41 Adam Williamson 2017-05-13 00:03:24 UTC
I'm setting this back to ASSIGNED since it's an active issue again, though really this bug shouldn't have been reopened, I don't think, a new bug should've been filed...

Comment 42 Marian Csontos 2017-05-16 19:21:48 UTC
Vrato, the following patch was suggested as potential fix: https://sourceware.org/git/?p=lvm2.git;a=commit;h=e53454d6de07de56736303dd2157c3859f6fa848

However, that is changing GetManagedObjects and LookUpByLvmId calls from synchronous to asynchronous. How big is that a deal? Is it a change in API or are dbus libs somehow abstracting it away?

Comment 43 Vratislav Podzimek 2017-05-18 13:00:07 UTC
(In reply to Marian Csontos from comment #42)
> Vrato, the following patch was suggested as potential fix:
> https://sourceware.org/git/?p=lvm2.git;a=commit;
> h=e53454d6de07de56736303dd2157c3859f6fa848
> 
> However, that is changing GetManagedObjects and LookUpByLvmId calls from
> synchronous to asynchronous. How big is that a deal? Is it a change in API
> or are dbus libs somehow abstracting it away?

Shouldn't affect the means Anaconda->Blivet->libblockdev get data from the DBus API.

Comment 44 Marian Csontos 2017-05-18 13:05:45 UTC
Except of one bug in original, the backport was straightforward.

[Untested yet] scratch build:

https://koji.fedoraproject.org/koji/taskinfo?taskID=19605558

(sorry, for not using better "Release:"...)

Vrato, are you able to test it? IIUC this requires an update.img which last time I tried I failed to produce working one.

..may be I could do something in %pre script in kickstart. Sorry, out of time today...

Comment 45 Tony Asleson 2017-05-18 13:25:52 UTC
(In reply to Marian Csontos from comment #42)
> Vrato, the following patch was suggested as potential fix:
> https://sourceware.org/git/?p=lvm2.git;a=commit;
> h=e53454d6de07de56736303dd2157c3859f6fa848
> 
> However, that is changing GetManagedObjects and LookUpByLvmId calls from
> synchronous to asynchronous. How big is that a deal? Is it a change in API
> or are dbus libs somehow abstracting it away?

This change did not change the external behavior.  The dbus calls from the client are still synchronous, they block until the result is returned (if that was their previous mode of operation).  Only the implementation changed.  However, clients are free to block or not block on the response as needed for their use case and what client side library they are using and the functionality it provides.

Comment 46 Marian Csontos 2017-05-19 09:48:53 UTC
I already realized it would likely be fine as DBus is all about messaging, so the client must either block or use callbacks while waiting for the response. Thanks Tony for confirmation.

I will attach updates.img. And try to test it...

Comment 47 Marian Csontos 2017-05-19 09:53:32 UTC
Created attachment 1280350 [details]
updates

To test add the following to boot options: updates=https://mcsontos.fedorapeople.org/bz1348688/updates-bz1348688.img

Comment 48 Marian Csontos 2017-05-19 11:33:24 UTC
Tested. I now see the LV instead of PV only. Building a package.

Comment 49 Fedora Update System 2017-05-19 11:47:20 UTC
lvm2-2.02.168-6.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-dc0776cb70

Comment 50 Kyle Marek 2017-05-19 13:02:49 UTC
Should show up in next nightly, I imagine?

Comment 51 Adam Williamson 2017-05-19 14:44:56 UTC
No, nightlies are built from stable (they don't include updates-testing packages). If you can test with the updates.img and provide feedback on the update it'd be very helpful, thanks!

Comment 52 Fedora Update System 2017-05-19 21:11:12 UTC
lvm2-2.02.168-6.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-dc0776cb70

Comment 53 Adam Williamson 2017-06-01 17:48:22 UTC
Steve, can you re-test with Beta RC4 (Beta-1.4) and see if it works with that? Thanks!

https://dl.fedoraproject.org/pub/alt/stage/26_Beta-1.4/

Comment 54 Mike Ruckman 2017-06-01 18:50:32 UTC
From my testing, it seems to work fine with 1.4

Comment 55 Steve Bryant 2017-06-01 20:26:56 UTC
Using "Fedora-Workstation-netinst-x86_64-26_Beta-1.4.iso" I was able to unlock my LUKS-encrypted partition and access/use the logical volumes contained to install Fedora 26 Beta.

This was possible using both the "Custom" and "Advanced Custom" options on the "Installation Destination" panel.

Comment 56 Fedora Update System 2017-06-03 17:38:55 UTC
lvm2-2.02.168-6.fc26 has been pushed to the Fedora 26 stable repository. If problems still persist, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.