Bug 910326

Summary: Upgrade doesn't start for systems with multiple encrypted partitions and an old systemd package
Product: [Fedora] Fedora Reporter: Kamil Páral <kparal>
Component: fedup-dracutAssignee: Will Woods <wwoods>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 18CC: admin, anthony.horton, blentz, brandon.gardner, bugzilla, collura, dan, danny_archer, david.k.lam1, giallu, gilboad, hafflys, it08165, james.r.wyatt, just18, luka.kowalski, michal.ambroz, milan.dlauhy, mjc, mr.postbox, nikodll, pat, rebus, tflink, wwoods, xeno
Target Milestone: ---Keywords: CommonBugs
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
See Also: https://bugzilla.redhat.com/show_bug.cgi?id=896010
Whiteboard: https://fedoraproject.org/wiki/Common_F18_bugs#fedup-encrypted
Fixed In Version: fedup-0.7.3-5.fc17 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 896010 Environment:
Last Closed: 2013-05-14 23:30:28 EDT Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Attachments:
Description Flags
tty1 "stuck"
none
lsblk output before and after sshd start none

Description Kamil Páral 2013-02-12 05:51:47 EST
+++ This bug was initially created as a clone of Bug #896010 +++

Description of problem:
This was reported by 'nyloc' on #fedora-qa, I reproduced it. I installed clean Fedora 17 on this disk layout:

vda1: /boot
vda2: / (encrypted)
vda3: swap (encrypted)
vdb1: /data (encrypted)

All standard partitions, no LVM. The password is the same for both systems.

After I finished running fedup-cli and rebooted, the system asks for disk password and then hangs. nyloc reported he received a maintenance shell after a timeout, I didn't, but that probably doesn't matter much.

This is the System Upgrade boot line from grub:

linux	/vmlinuz-fedup root=/dev/mapper/luks-f1cab7e0-fae8-4a00-94b9-c5d3976d6031 ro rd.luks.uuid=luks-80aa76ad-d5a3-4bfa-b8cd-7f70fe8379a0 rd.lvm=0 rd.dm=0 SYSFONT=True rd.luks.uuid=luks-f1cab7e0-fae8-4a00-94b9-c5d3976d6031  KEYTABLE=us rd.md=0 LANG=en_US.UTF-8 rhgb quiet upgrade systemd.unit=system-upgrade.target plymouth.splash=fedup

It looks fine.

This is the System Upgrade boot output:

> dracut-cmdline[50]: Warning: Kernel command line option 'KEYTABLE' is deprecated, use 'vconsole.keymap' instead.
> dracut-cmdline[50]: Warning: Kernel command line option 'SYSFONT' is deprecated, use 'vconsole.font' instead.
> [    3.114503] systemd-cryptsetup-generator[265]: Failed to create unit file: File exists
> [  OK  ] Found device /dev/disk/by-uuid/80aa76ad-d5a3-4bfa-b8cd-7f70fe8379a0.
>          Starting Cryptography Setup for luks-80aa76ad-d5a3-4...7f70fe8379a0...
> [  OK  ] Found device /dev/disk/by-uuid/f1cab7e0-fae8-4a00-94b9-c5d3976d6031.
>          Starting Cryptography Setup for luks-f1cab7e0-fae8-4...c5d3976d6031...
>          Expecting device dev-mapper-luks\x2d80aa76ad\x2dd5a3...379a0.device...
>          Expecting device dev-mapper-luks\x2df1cab7e0\x2dfae8...d6031.device...
> [  OK  ] Started Show Plymouth Boot Screen.
>          Starting Forward Password Requests to Plymouth...
> [  OK  ] Started Forward Password Requests to Plymouth.
> 
> Please enter passphrase for disk luks-f1cab7e0-fae8-4a00-94b9-c5d3976d6031!:**********
> [  OK  ] Started Cryptography Setup for luks-f1cab7e0-fae8-4a...9-c5d3976d6031.
> [  OK  ] Found device /dev/mapper/luks-f1cab7e0-fae8-4a00-94b9-c5d3976d6031.
> [  OK  ] Started Cryptography Setup for luks-80aa76ad-d5a3-4b...d-7f70fe8379a0.
> [  OK  ] Reached target Local File Systems.
> [  OK  ] Reached target Encrypted Volumes.
> [  OK  ] Reached target System Initialization.
> [  OK  ] Reached target Basic System.              <------ HANGS HERE

Boot with rd.debug is attached.

System reboots if Ctrl+Alt+Del is hit, so it's not dead.

Version-Release number of selected component (if applicable):
fedup-dracut of unknown version taken from development/18 tree. (Hey Will, why it is not mentioned anywhere, like in --debuglog?)

How reproducible:
with nyloc and my confirmation, seems like every time

Steps to Reproduce:
1. follow description

--- Additional comment from Kamil Páral on 2013-01-16 14:15:59 CET ---

Created attachment 679569 [details]
fedup.log

--- Additional comment from Kamil Páral on 2013-01-16 14:16:03 CET ---

Created attachment 679570 [details]
grub.cfg

--- Additional comment from Kamil Páral on 2013-01-16 14:16:09 CET ---

Created attachment 679571 [details]
boot.debug.log

--- Additional comment from Kamil Páral on 2013-01-18 10:55:21 CET ---

There are probably several issues combined in this report. Adding selinux=0 didn't help me (nothing changed) with the original problem described in comment 0.

--- Additional comment from Kamil Páral on 2013-02-01 10:04:31 CET ---

fedup-0.7.3-0.git20130128.fc17 doesn't fix the problem mentioned in comment 0.

--- Additional comment from Will Woods on 2013-02-11 20:44:25 CET ---

If adding 'enforcing=0' and removing 'rhgb' doesn't fix your problem, please file a new bug so it doesn't get lost!
Comment 1 Will Woods 2013-02-12 13:23:51 EST
The dracut log isn't going to be helpful here; the problem happens when we're back in the old (F17) system and trying to mount disks.

You could add 'rd.upgrade.debugshell' to the boot args and use the shell on VT2 to figure out what's going on with the stuck system. I'm guessing it's waiting for the LUKS partition to be unlocked; since that's stuck, you could just wait a few minutes and you should get an emergency shell from systemd.

Either way, can you get the syslog/journalctl output from the system when it's stuck and attach that?
Comment 2 Kamil Páral 2013-02-13 07:31:31 EST
> I'm guessing it's
> waiting for the LUKS partition to be unlocked; since that's stuck, you could
> just wait a few minutes and you should get an emergency shell from systemd.

No, there is no timeout happening, no emergency shell. The system is waiting for something, but it is not completely stuck, Ctrl+Alt+Del works.

I used rd.upgrade.debugshell, but I don't how to access any logs. There are no commands available, 'ls' is not there, 'less' is not there, nothing. 'help' prints out some internal bash commands, but those doesn't help get the job done.

Can you please try to reproduce the problem yourself? It should be 100% reproducible using the same disk layout. Or I can upload the whole VM image somewhere, if you prefer.
Comment 3 Will Woods 2013-02-14 10:59:08 EST
(In reply to comment #2)

> I used rd.upgrade.debugshell, but I don't how to access any logs. There are
> no commands available, 'ls' is not there, 'less' is not there, nothing.
> 'help' prints out some internal bash commands, but those doesn't help get
> the job done.

The shell is in the wrong root. Exit and it will restart in your current root.
Comment 4 Will Woods 2013-02-14 13:55:11 EST
I set up a VM with the following partitions:

  /dev/vda1: /boot
  /dev/vda2: swap (encrypted)
  /dev/vda3: / (encrypted)
  /dev/vdb1: /home (encrypted)

I installed fedup-0.7.3-0.git20130128.fc17.noarch and ran it. It completed successfully.

The 'fedup' boot entry had "enforcing=0" in it already; I removed "rhgb" *and* "plymouth.splash=fedup".

The upgrade is now at 56% complete, so I consider it working.

Can you try removing "rhgb" *and* "plymouth.splash=fedup" and see if that works? If so, this is very likely a dupe of bug 894242.
Comment 5 Kamil Páral 2013-02-18 05:45:50 EST
When I remove rhgb and plymouth.splash=fedup there is no change, still stuck.

If I add rd.upgrade.debugshell and exit the shell once, I really appear in a real filesystem and can use real system commands, as you said. The filesystem is read only and the last line in /var/log/messages is from Feb 1st. That is probably the last day I ran the original F17 system. Also judging from its contents (and contents of other log files), there are not actual, but really from Feb 1st.

I also found /system-upgrade-root/var/log, but that doesn't contain any files.

journalctl says "Failed to iterate through journal: No such file or directory".

Looking as lsblk output, vda2 and vda3 was decrypted, but vdb1 was not. An interesting thing is that if I try to start sshd, a prompt is printed out for password for vdb1 (but I can't input anything) and after a second vdb1 gets decrypted. See screenshot. Once that happens, two additional lines are printed on tty1 from systemd-fsck about vda1 and vdb1 being clean.

I think you should have mirrored my disk layout precisely. Ping me on IRC to help me debug it.
Comment 6 Kamil Páral 2013-02-18 05:46:42 EST
Created attachment 698821 [details]
tty1 "stuck"
Comment 7 Kamil Páral 2013-02-18 05:47:17 EST
Created attachment 698822 [details]
lsblk output before and after sshd start
Comment 8 Kamil Páral 2013-02-18 08:17:15 EST
Will, I tried completely the same disk layout as you have in comment 4, the same fedup version, I removed the same grub options, and it is stuck! Now tell me, what the hell is wrong with me?! :-)

I was working with F17 x86_64 Desktop Live install, no updates. Maybe you were working with a different architecture, or minimal install? We have to find the difference.
Comment 9 Will Woods 2013-02-18 11:26:45 EST
Your system isn't updated? That's very likely the problem. 

Try updating systemd; if that doesn't work, try updating selinux-policy; if neither of those works, try a full update.

If updating makes the upgrade work, we should file that as a new bug.

If that turns out to be the case, could you note the version(s) of each that don't work? I should probably make fedup request those package versions.
Comment 10 Kamil Páral 2013-02-18 12:33:45 EST
Great call, Will. Systemd update fixed the problem.

Systemd before: systemd-44-8.fc17.x86_64
Systemd after: systemd-44-23.fc17.x86_64

I wonder, wouldn't it be better if fedup insisted that there must be no system updates available before it sets up the system for upgrade? This time it was systemd, next time it might be something else.
Comment 11 Fedora Update System 2013-03-15 19:35:16 EDT
fedup-0.7.3-1.fc17 has been submitted as an update for Fedora 17.
https://admin.fedoraproject.org/updates/fedup-0.7.3-1.fc17
Comment 12 Fedora Update System 2013-03-16 20:55:20 EDT
Package fedup-0.7.3-1.fc17:
* should fix your issue,
* was pushed to the Fedora 17 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing fedup-0.7.3-1.fc17'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2013-3956/fedup-0.7.3-1.fc17
then log in and leave karma (feedback).
Comment 13 Fedora Update System 2013-04-30 10:52:12 EDT
fedup-0.7.3-3.fc18 has been submitted as an update for Fedora 18.
https://admin.fedoraproject.org/updates/fedup-0.7.3-3.fc18
Comment 14 Fedora Update System 2013-04-30 10:56:08 EDT
fedup-0.7.3-3.fc17 has been submitted as an update for Fedora 17.
https://admin.fedoraproject.org/updates/fedup-0.7.3-3.fc17
Comment 15 Fedora Update System 2013-04-30 11:38:22 EDT
fedup-0.7.3-4.fc17 has been submitted as an update for Fedora 17.
https://admin.fedoraproject.org/updates/fedup-0.7.3-4.fc17
Comment 16 Fedora Update System 2013-04-30 11:39:08 EDT
fedup-0.7.3-4.fc18 has been submitted as an update for Fedora 18.
https://admin.fedoraproject.org/updates/fedup-0.7.3-4.fc18
Comment 17 Fedora Update System 2013-04-30 11:40:07 EDT
fedup-0.7.3-4.fc19 has been submitted as an update for Fedora 19.
https://admin.fedoraproject.org/updates/fedup-0.7.3-4.fc19
Comment 18 Fedora Update System 2013-05-14 23:30:28 EDT
fedup-0.7.3-4.fc18 has been pushed to the Fedora 18 stable repository.  If problems still persist, please make note of it in this bug report.
Comment 19 Chris Schanzle 2013-05-24 11:21:53 EDT
Unfortunately, fedup 0.7.3-4 didn't work for me with a fully updated system.  Wouldn't boot into the anaconda installer.

$ rpm -q fedup systemd
fedup-0.7.3-4.fc17.noarch
systemd-44-24.fc17.x86_64
systemd-44-24.fc17.i686

$ sudo blkid
/dev/sda2: UUID="d8227018-346d-47b5-a74e-4eebbab677a1" TYPE="ext4" 
/dev/sda3: UUID="59778677-f7e9-4d4d-9bbd-9289542a67bc" TYPE="crypto_LUKS" 
/dev/sda4: UUID="d6f87459-69cf-4646-9a95-bf82d074ad3f" TYPE="crypto_LUKS" 
/dev/sda5: UUID="6787bfb0-f594-4270-a539-0af35470c8bc" TYPE="crypto_LUKS" 
/dev/mapper/luks-59778677-f7e9-4d4d-9bbd-9289542a67bc: UUID="ead53e81-5780-46c6-ae26-079e181977ed" TYPE="ext4" 
/dev/mapper/luks-d6f87459-69cf-4646-9a95-bf82d074ad3f: UUID="be7adbf3-b64f-43af-ba11-ebe211ee9ca8" TYPE="ext4" 
/dev/mapper/luks-6787bfb0-f594-4270-a539-0af35470c8bc: UUID="35c3d5f9-5651-4371-8527-541d3d668486" TYPE="swap" 

I'm sorry this is thin on the info, but I'd be happy to provide more upon request.
Comment 20 Will Woods 2013-05-24 14:39:53 EDT
It's not supposed to boot into anaconda. 

It boots into the upgrade tool, which basically just shows a progress meter (or log text) while it upgrades your system.
Comment 21 Chris Schanzle 2013-05-25 09:13:14 EDT
Please excuse my ignorance with fedup upgrades, I usually do clean installs.  It doesn't start the upgrade tool.  Booting without rhgb, after I enter my passphrase, get the Welcome to Fedora 17, starting services, noted after Mounting Media Directory:

         Expecting device dev-disk-by\x2duuid-d8227018\x2d346d\x2d47b5\x2da74e\x2d4eebbab677a1.device...
         Expecting device dev-disk-by\x2duuid-d6f87459\x2d69cf\x2d4646\x2d9a95\x2dbf82d074ad3f.device...
         Expecting device dev-mapper-luks\x2dd6f87459\x2d69cf\x2d4646\x2d9a95\x2dbf82d074ad3f.device...

Long pause (~1min) after the first line below, then [transcribing from photo]

Started Initialize storage subsystems (RAID, LVM, etc.)
Timed out waiting for device dev-mapper-luks\0x2dd6f87459\x2d4646\x2d9a95\x2dbf82d074ad3f.device.
Dependency failed for Cryptography Setup for luks-d6f87459-69cf-4646-9a95-bf82d074ad3f
Dependency failed for Encrypted Volumes.
Dependency failed for /home.
Dependency failed for Local File Systems.
Welcome to emergency mode.  Use "systemctl default" or ^D to enter default mode.

Re-open or start new bug?
Comment 22 Gilboa Davara 2013-05-26 11:24:28 EDT
Not sure if I should reopen this BZ, but latest fedup shows the same issue as previous releases: On a fully updated F17 the luks prompt is never shown and after a (long) wait, the emergency prompt is shown.

Should I reopen this BZ?

- Gilboa
Comment 23 Fedora Update System 2013-06-07 12:43:49 EDT
fedup-0.7.3-5.fc17 has been submitted as an update for Fedora 17.
https://admin.fedoraproject.org/updates/fedup-0.7.3-5.fc17
Comment 24 Fedora Update System 2013-06-07 23:35:06 EDT
fedup-0.7.3-4.fc19 has been pushed to the Fedora 19 stable repository.  If problems still persist, please make note of it in this bug report.
Comment 25 Fedora Update System 2013-06-23 01:57:06 EDT
fedup-0.7.3-5.fc17 has been pushed to the Fedora 17 stable repository.  If problems still persist, please make note of it in this bug report.