Created attachment 838376 [details]
compressed fedup logs from the upgrade preparation

Created attachment 838377 [details]
grub2 configuration

Please advise what other logs are needed, I'd attach a journalctl -a -b when in the upgrade boot, and also blkid/lsblk -f. Anything else?

Created attachment 838379 [details]
compressed /var/log/upgrade.log

This what is displayed when booting the System Upgrade, last message shown in red, (alternatingly on the same line):

(1 of 2) A start job is running for dev-mapper-luks\...775aefd3.device
(2 of 2) A start job is running for Cryptography set 26-3146775aefd3

Boot does not continue any further.

Output from lsblk -f (/home partition is not mounted/missing, but LV exists):

# cat lsblk_-f.txt 
NAME                                              FSTYPE      LABEL UUID                                   MOUNTPOINT
sda                                                                                                        
├─sda1                                            ext4              5578523e-b1ea-47ef-83c3-777efc2fbd13   
└─sda2                                            crypto_LUKS       7dc71d69-0c5d-4132-a4fd-776db8275242   
sdb                                                                                                        
├─sdb1                                            ext4              a5a1e9ca-e533-4884-ab76-d59ac4a9a472   /boot
└─sdb2                                            crypto_LUKS       1050af2f-9b9b-4282-9a3f-517acc693819   
  └─luks-1050af2f-9b9b-4282-9a3f-517acc693819     LVM2_member       6j1VAp-vAGm-FNK3-wBF7-n6Ei-ARTD-i7Te6q 
    ├─ocfedora19-swap                             crypto_LUKS       08e0c033-95f0-4650-b3e8-67b71778c2e3   
    │ └─luks-08e0c033-95f0-4650-b3e8-67b71778c2e3 swap              f1172e8a-b88c-4a10-99ab-3b0448e63a09   [SWAP]
    ├─ocfedora19-root                             crypto_LUKS       02288c2d-284e-4f21-8dd0-4b35bc8042db   
    │ └─luks-02288c2d-284e-4f21-8dd0-4b35bc8042db ext4              1ab516fc-117f-4cda-a8d4-432032629f85   /
    └─ocfedora19-home                             crypto_LUKS       a963b5df-5a18-46c9-b126-3146775aefd3

What's missing is luks-a963b5df-5a18-46c9-b126-3146775aefd3, my /home partition.
Seems it wants to do an fsck on /home but fails as it's not mounted. /home FS is clean per fsck.

# systemctl | grep luks
dev-mapper-luks\x2da963b5df\x2d5a18\x2d46c9\x2db126\x2d3146775aefd3.device                  loaded inactive   dead      start dev-mapper-luks\x2da963b5df\x2d5a18\x2d46c9\x2db126\x2d3146775aefd3.device
systemd-cryptsetup@luks\x2d02288c2d\x2d284e\x2d4f21\x2d8dd0\x2d4b35bc8042db.service         loaded active     exited          Cryptography Setup for luks-02288c2d-284e-4f21-8dd0-4b35bc8042db
systemd-cryptsetup@luks\x2d08e0c033\x2d95f0\x2d4650\x2db3e8\x2d67b71778c2e3.service         loaded active     exited          Cryptography Setup for luks-08e0c033-95f0-4650-b3e8-67b71778c2e3
systemd-cryptsetup@luks\x2d1050af2f\x2d9b9b\x2d4282\x2d9a3f\x2d517acc693819.service         loaded active     exited          Cryptography Setup for luks-1050af2f-9b9b-4282-9a3f-517acc693819
systemd-cryptsetup@luks\x2da963b5df\x2d5a18\x2d46c9\x2db126\x2d3146775aefd3.service         loaded activating start     start Cryptography Setup for luks-a963b5df-5a18-46c9-b126-3146775aefd3
systemd-fsck@dev-mapper-luks\x2da963b5df\x2d5a18\x2d46c9\x2db126\x2d3146775aefd3.service    loaded inactive   dead      start File System Check on /dev/mapper/luks-a963b5df-5a18-46c9-b126-3146775aefd3

Created attachment 838413 [details]
output of systemctl in fedup boot

Created attachment 838414 [details]
output from journalctl -a -b in fedup boot

General question:

As booting with the fc19 kernel is totally fine into the upgraded system, can I just continue e.g. via
fedup --reset-bootloader; fedup --clean

or should I run "fedup-cli --network 20" again?

I'm trying to reproduce this now. F19 Guided Partitioning defaults (LVM+ext4) with encryption does not encrypt /boot, which is placed on its own ext4 partition. The remaining disk space is a single encrypted PV from which LV's for /, /home, and swap are created. So until we figure out why you aren't given an option to luksOpen when booting the fedup kernel/initramfs, then none of those LV's is visible or mountable and thus not upgradeable.

If you have a cell phone camera to photograph the boot failure messages, when choosing the Fedup option from the GRUB menu, that might be helpful.

OK I think I know what's going on, but I don't have a work around for it yet. You have double encryption occurring so chances are you will want to reinstall because this is going to give you quite the performance  hit.

Here's a cleaned up version of what you posted:

sdb                                               disk  
├─sdb1                                            part  /boot
└─sdb2                                            part  
  └─luks-1050af2f-9b9b-4282-9a3f-517acc693819     crypt 
    ├─ocfedora19-swap                             lvm   
    │ └─luks-08e0c033-95f0-4650-b3e8-67b71778c2e3 crypt [SWAP]
    ├─ocfedora19-root                             lvm   
    │ └─luks-02288c2d-284e-4f21-8dd0-4b35bc8042db crypt /
    └─ocfedora19-home                             lvm   
      └─luks-a963b5df-5a18-46c9-b126-3146775aefd3 crypt /home
      
This is an F19 Guided Partitioning only, Encrypt option checked. I did not go to customize partitioning (the Manual Partitioning screen).
      
vda                                           disk  
├─vda1                                        part  /boot
└─vda2                                        part  
  └─luks-92bf82da-0e50-4a65-99cf-7fe012de7e4b crypt 
    ├─fedora-swap                             [SWAP]
    ├─fedora-root                             /
    └─fedora-home                             /home

As you can see, vda2 is encrypted, which then becomes a PV/VG from which swap, root, and home are created.

But in your case, you have both an encrypted PV, and then you also have each LV separately encrypted. I think this can happen if you check encryption on the Installation Options dialog, then go into custom, and for each mount point you also then check the Encrypt checkbox.

The installer shouldn't allow this but I think that's what's happening. So I'd say you really want to start over because double encryption is a big performance hit, and I'm skeptical that the upgrade tool should support this, even though it was allowed to be created. Does this make sense?

(In reply to Chris Murphy from comment #9)
> I'm trying to reproduce this now. F19 Guided Partitioning defaults
> (LVM+ext4) with encryption does not encrypt /boot, which is placed on its
> own ext4 partition. The remaining disk space is a single encrypted PV from
> which LV's for /, /home, and swap are created. So until we figure out why
> you aren't given an option to luksOpen when booting the fedup
> kernel/initramfs, then none of those LV's is visible or mountable and thus
> not upgradeable.

Well, I'm asked for the password, as there's only one password for the PV.
And / and swap are mounted successfully, so that's not the problem.


> If you have a cell phone camera to photograph the boot failure messages,
> when choosing the Fedup option from the GRUB menu, that might be helpful.


Well, there's no failure message, only message where boot waits I've posted in #c5, but I can also take a screenshot if you think that's helpful.

The same password was used to create all four LUKS volumes, which is why they all get luksOpen'd. However, the reality is, your setup is double encrypting everything which isn't necessary, overly complicates the setup, and will definitely slow things down. I personally would reinstall and ensure that you aren't both encrypting the PV/VG as well as each individual LV. Pick one or the other.

I'm not totally sure but I don't think /home is needed for this update so you could try to comment out /home in fstab just for the update, and then remove the comment after the update. This might allow the upgrade to proceed.

The upgrade process should not need to touch /home, indeed.

(In reply to Chris Murphy from comment #10)
> OK I think I know what's going on, but I don't have a work around for it
> yet. You have double encryption occurring so chances are you will want to
> reinstall because this is going to give you quite the performance  hit.
> 
> sdb                                               disk  
> ├─sdb1                                            part  /boot
> └─sdb2                                            part  
>   └─luks-1050af2f-9b9b-4282-9a3f-517acc693819     crypt 
>     ├─ocfedora19-swap                             lvm   
>     │ └─luks-08e0c033-95f0-4650-b3e8-67b71778c2e3 crypt [SWAP]
>     ├─ocfedora19-root                             lvm   
>     │ └─luks-02288c2d-284e-4f21-8dd0-4b35bc8042db crypt /
>     └─ocfedora19-home                             lvm   
>       └─luks-a963b5df-5a18-46c9-b126-3146775aefd3 crypt /home
>       
> This is an F19 Guided Partitioning only, Encrypt option checked. I did not
> go to customize partitioning (the Manual Partitioning screen).
>       
> vda                                           disk  
> ├─vda1                                        part  /boot
> └─vda2                                        part  
>   └─luks-92bf82da-0e50-4a65-99cf-7fe012de7e4b crypt 
>     ├─fedora-swap                             [SWAP]
>     ├─fedora-root                             /
>     └─fedora-home                             /home
> 
> As you can see, vda2 is encrypted, which then becomes a PV/VG from which
> swap, root, and home are created.
> 
> But in your case, you have both an encrypted PV, and then you also have each
> LV separately encrypted. I think this can happen if you check encryption on
> the Installation Options dialog, then go into custom, and for each mount
> point you also then check the Encrypt checkbox.

Nope, I've only deleted all volumes from the device, and then let installer create everything, and ticked encryption only once, entered the passphrase and 
that's pretty much it.

> 
> The installer shouldn't allow this but I think that's what's happening. So
> I'd say you really want to start over because double encryption is a big
> performance hit, and I'm skeptical that the upgrade tool should support
> this, even though it was allowed to be created. Does this make sense?

Are you sure this is double encryption? I've not problem with trying to reproduce this, as I still need to test some stuff with a clean install, and will probably compare with a fresh F20 install.

Also didn't notice anything related to disk performance or cpu time, still really good but will look out for that.

(In reply to Adam Williamson from comment #13)
> The upgrade process should not need to touch /home, indeed.

Well, then only thing which remains for the upgrade is that fedup cleans the bootloader entry, and the new kernel gets added, no? So my question from https://bugzilla.redhat.com/show_bug.cgi?id=1044551#c8 still stands.

What does the fedup process normally look like? Download all packages & intrd, add new bootloader entry, reboot, do the upgrade, reboot and do the post-upgrade/cleanup?

Then I'd be currently in last phase, seems fedup just can't continue due to the issue with luks. I'd first try to get the upgrade completed, and then will do a reinstall and take a look at the partitioning.

florian: I haven't been following your case in detail, I was just providing some info to cmurf, who seems to be handling it fine.

AFAIK, fedup cleans up at the end of the upgrade stage, not on the final boot back to the installed system.

(In reply to Florian Hubold from comment #14)
> Nope, I've only deleted all volumes from the device, and then let installer
> create everything, and ticked encryption only once, entered the passphrase
> and that's pretty much it.

"Deleted all volumes" implies a previous installation. If that previous installation had an encrypted PV/VG, it only appears in the Volume Group Options dialog by clicking on the Modify button to the right of the VG name pop-up menu. By deleting volumes, you are not deleting the existing encrypted VG, it remains. By then ticking encryption again for each LV, you're doubling the encryption. I've reproduced this in the F19 installer and I think it's fixed in the F20 installer so users can't created nested LUKs volumes. Very clearly your lsblk shows you have nested LUKs volumes.

See bug 1030416, bug 1038969, and bug 1040720.

> Are you sure this is double encryption?

Yes.

> Also didn't notice anything related to disk performance or cpu time, still
> really good but will look out for that.

If the CPU has AES-NI it's possible you won't notice except under heavier read/write workloads.

(In reply to Chris Murphy from comment #17)
> (In reply to Florian Hubold from comment #14)
> By deleting volumes, you are not deleting the existing
> encrypted VG, it remains. By then ticking encryption again for each LV,
> you're doubling the encryption. I've reproduced this in the F19 installer
> and I think it's fixed in the F20 installer so users can't created nested
> LUKs volumes. Very clearly your lsblk shows you have nested LUKs volumes.
> 
> See bug 1030416, bug 1038969, and bug 1040720.
> 
> > Are you sure this is double encryption?
> 
> Yes.

Yep, verified. As this one of my first Fedora installs in recent time, I've ticket encryption for each LV in custom partitioning, as the normal complete automatic method was rather un-obvious. I've used it now for a clean F19 install and now it looks normal, as you posted already:

$ lsblk
[...]
sdb                                             8:16   0 111,8G  0 disk  
├─sdb1                                          8:17   0   500M  0 part  /boot
└─sdb2                                          8:18   0 111,3G  0 part  
  └─luks-32f7499c-fdaa-4c2f-930b-8f1580f01231 253:0    0 111,3G  0 crypt 
    ├─fedora-swap                             253:1    0   7,9G  0 lvm   [SWAP]
    ├─fedora-root                             253:2    0    50G  0 lvm   /
    └─fedora-home                             253:3    0  53,4G  0 lvm   /home


> 
> > Also didn't notice anything related to disk performance or cpu time, still
> > really good but will look out for that.
> 
> If the CPU has AES-NI it's possible you won't notice except under heavier
> read/write workloads.

Yep, it has and it's writing onto pretty fast SSD, so you're right there, probably didn't notice during normal operation. But it could explain some of the micro-freezes I've seen during big copy/move operations.


So overall the double encryption is probably the cause for the issues with mounting swap and /home during or post-upgrade. Just completed fedup upgrade with a fresh installation and above mentioned LVM/luks setup which worked flawlessly, as one would expect it.



@Adam: My proposal would be to make sure that the installer will show a warning to the user if he enables this "double encryption" and it should also be documented somewhere. (Maybe it is already, I've just not read the documentation :) )


Also for the installer and partitioning choice in general, maybe it can be changed to something which is more obvious, similar to the one from Mandriva's diskdrake which you probably still remember: http://doc.mageia.org/installer/3/en/content/doPartitionDisks.html

Please feel free to close this report.

We've patched it in F20 so this kind of double-encryption shouldn't be possible to achieve any more (well, maybe if you screw around enough you could make it happen, but it should be much more difficult to do it *inadvertently*).

I'm not sure the report needs to be closed, I mean, I guess fedup still could theoretically work in such a setup. But it's probably low priority. I guess it's up to Will.

(In reply to Florian Hubold from comment #18)
> Yep, verified. As this one of my first Fedora installs in recent time, I've
> ticket encryption for each LV in custom partitioning, as the normal complete
> automatic method was rather un-obvious.

Agreed.

> Yep, it has and it's writing onto pretty fast SSD, so you're right there,
> probably didn't notice during normal operation. But it could explain some of
> the micro-freezes I've seen during big copy/move operations.

It's possible, since in effect there's multiple device mapper block devices: luks, LV, luks, and then the physical block device. Another possibility is the use of discard mount option for certain SSDs, which enables TRIM commands. This isn't the Fedora default so if this is applicable it's something you'd have to explicitly change in fstab.

> So overall the double encryption is probably the cause for the issues with
> mounting swap and /home during or post-upgrade.

Maybe. I'm finding other issues like bug 1045168. I'm about to test variations of /home not on rootfs and see. I have one case for sure where /home on raid1 causes fedup boot environment delays, it eventually times out and completes the upgrade. Other combinations of /home not on rootfs may fail, not sure yet.

> Just completed fedup upgrade
> with a fresh installation and above mentioned LVM/luks setup which worked
> flawlessly, as one would expect it.

Great!

Hello Chris, Williamson,

I am also facing the same problem i.e. "Comment 5" of Florian Hubold.

May I request you to please confirm; below output of lsblk command is also looking to be "double encryption"? I did manua l partition during Fedora 18 and then upgraded to Fedora 19 via fedup in past.

Output from lsblk -f

$ lsblk -f 
NAME                                          FSTYPE      LABEL       UUID                                 MOUNTPOINT
sda                                                                                                        
├─sda1                                        vfat        DellUtility 3030-3030                            
├─sda2                                        ntfs        RECOVERY    80E637BAE637AF72                     
├─sda3                                        ntfs        OS          907C39657C39476E                     
├─sda4                                                                                                     
├─sda5                                        ntfs        Win7Backup  319EF2654F2ECDA4                     
├─sda6                                        ntfs        Shared      32E64956017591FA                     
├─sda7                                        ext4                    1451240e-89a8-4239-8208-8fa5154e34f8 /boot
├─sda8                                        crypto_LUKS             0ffed240-c9aa-41eb-9b6d-b3ea2747c47f 
│ └─luks-0ffed240-c9aa-41eb-9b6d-b3ea2747c47f ext4                    58eecd3f-171f-4b80-af62-0e44cde2547d /home
├─sda9                                        crypto_LUKS             689c2139-60c6-478e-b16b-ce3a194dd9f9 
│ └─luks-689c2139-60c6-478e-b16b-ce3a194dd9f9 swap                    8ecd2dc8-3efa-4584-8f58-fcf85264dfe8 [SWAP]
└─sda10                                       crypto_LUKS             be850422-7d28-4e1e-a4ee-2cfa4dda8c3e 
  └─luks-be850422-7d28-4e1e-a4ee-2cfa4dda8c3e ext4                    67607e47-ecc0-4bf5-a0df-9c7dc53ffdfb /
sdb                                                                                                        
└─sdb1                                        vfat        LIVE        2CB6-7CBE                            /media/HAMA-8GB
sr0  

Waiting for your reply.

Br
Ashish

(In reply to Ashish from comment #21)
>below output of lsblk command is also
> looking to be "double encryption"? 

It's not double encrypted. You're also not using LVM.

AFAICT all of these problems are the same root cause: multiple LUKS partitions using the same password. That's bug 1045864. See that bug for details and workarounds.

*** This bug has been marked as a duplicate of bug 1045864 ***

Hello Chris, Will,updated 

1.) I was able to proceed/boot in F20 by putting enforcing=0 parameter in grub start menu but even after booting the F20 entry is not added automatically and hence when I reboot I have to again to do the same i.e. add enforcing=0 parameter in grub start menu every time.

2.) Do you suggest to re-install the F20 and use only LVM (automatic partitioning) and not use manual partitioning?

3.) One more issue I am observing here is WiFi seems to be disabled.

root@localhost ashish# uname -a
Linux localhost.localdomain 3.11.10-301.fc20.x86_64 #1 SMP Thu Dec 5 14:01:17 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

root@localhost ashish# lspci | grep -i network
03:00.0 Network controller: Intel Corporation Centrino Wireless-N 1000 [Condor Peak]

When I do yum update, it says "No packages marked for update".
The point is that my F20 kernel is "3.11.10-301" while F19 old kernel is kernel-3.13.5-101.fc19.x86_64.

I tried to find in google but no success.

Note: I think I need to fresh install :( F20 to get rid of these issues.

Br
Ashish

(In reply to Ashish from comment #24)
> Hello Chris, Will,updated 
> 
> 1.) I was able to proceed/boot in F20 by putting enforcing=0 parameter in
> grub start menu but even after booting the F20 entry is not added
> automatically and hence when I reboot I have to again to do the same i.e.
> add enforcing=0 parameter in grub start menu every time.

Unless you're booting the 'System Upgrade (fedup)' boot item, you shouldn't need enforcing=0. That's a totally different problem, though.

> 2.) Do you suggest to re-install the F20 and use only LVM (automatic
> partitioning) and not use manual partitioning?

No, this should work as expected - once the bug in fedup/systemd gets fixed.

> 3.) One more issue I am observing here is WiFi seems to be disabled.

This is a totally unrelated problem; I don't maintain the kernel or the wireless drivers so I can't help here.

> When I do yum update, it says "No packages marked for update".
> The point is that my F20 kernel is "3.11.10-301" while F19 old kernel is
> kernel-3.13.5-101.fc19.x86_64.

Yes. kernel-3.13.5-101.fc19.x86_64 is newer than the F20 kernel, so there's no need to update. You might try 'yum distro-sync' or 'yum reinstall kernel'.


A gentle reminder: Bugzilla is not a support forum. Please don't throw unrelated problems into closed bug reports - it just makes it harder to trace and fix the original bug.