Bug 1077961 - System fails to activate random VGs during boot.
Summary: System fails to activate random VGs during boot.
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: lvm2
Version: 20
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: ---
Assignee: Peter Rajnoha
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-03-19 00:43 UTC by David
Modified: 2014-04-20 01:28 UTC (History)
12 users (show)

Fixed In Version: lvm2-2.02.106-1.fc20
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2014-04-20 01:28:41 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)
output of journalctl -b at emergency shel (527.50 KB, text/x-vhdl)
2014-03-19 11:05 UTC, David
no flags Details
output of lsblk at emergency shell (6.97 KB, text/plain)
2014-03-19 11:06 UTC, David
no flags Details
lvmdump at emergency shell (114.46 KB, application/x-gzip)
2014-03-19 11:06 UTC, David
no flags Details
output of lsblk at default run level (8.27 KB, text/plain)
2014-03-19 11:07 UTC, David
no flags Details
lvmdump at default run level (116.34 KB, application/x-gzip)
2014-03-19 11:08 UTC, David
no flags Details

Description David 2014-03-19 00:43:43 UTC
Description of problem:
while booting the system fails to activate random VGs. System drops to emergency shell and fails to boot. 

Version-Release number of selected component (if applicable):
 2.02.103(2) (2013-10-04)

How reproducible:
Everytime

Steps to Reproduce:
1. Boot system
2.
3.

Actual results:
System fails to activate VGs and mount filesystems, fails to boot. 

Expected results:
On boot system activates all VGs, LVs, and mounts filesystems. Boots to default run mode. 

Additional info:
getting messages like this in journal:
Mar 18 09:08:11 buttercup.redzone.thehillhome.net pvscan[967]: Request to update PV 15nr2D-jDIM-Rytj-pYJl-Qrhv-tR0z-wCg57t in lvmetad gave respo
Mar 18 09:08:11 buttercup.redzone.thehillhome.net pvscan[967]: Update of lvmetad failed. This is a serious problem.
Mar 18 09:08:11 buttercup.redzone.thehillhome.net pvscan[967]: It is strongly recommended that you restart lvmetad immediately.
Mar 18 09:08:11 buttercup.redzone.thehillhome.net systemd[1]: lvm2-pvscan@8:4.service: main process exited, code=exited, status=5/NOTINSSTALLED
Mar 18 09:08:11 buttercup.redzone.thehillhome.net systemd[1]: Failed to start LVM2 PV scan on device 8:4.
Mar 18 09:08:11 buttercup.redzone.thehillhome.net systemd[1]: Unit lvm2-pvscan@8:4.service entered failed state.

When in emergency shell, I can manually activate VGs. File systems mount and I can proceed to default run mode.

Comment 1 Peter Rajnoha 2014-03-19 08:08:41 UTC
Please, take these steps and provide following information for analysis:

  - add "systemd.log_level=debug systemd.log_target=kmsg log_buf_len=1M" to kernel command line
  - boot your system, when dropped to shell after failure, run "journalctl -b" and save the output and attach it here
  - save the output from "lsblk" and "lvmdump -ul" command and attach it here too
  - then manually activate the volumes and when the system if fully started, run "lsblk" and "lvmdump -ul" again and attach the output

Thanks.

Comment 2 David 2014-03-19 11:05:01 UTC
Created attachment 876315 [details]
output of journalctl -b at emergency shel

Comment 3 David 2014-03-19 11:06:02 UTC
Created attachment 876316 [details]
output of lsblk at emergency shell

Comment 4 David 2014-03-19 11:06:42 UTC
Created attachment 876317 [details]
lvmdump at emergency shell

Comment 5 David 2014-03-19 11:07:30 UTC
Created attachment 876318 [details]
output of lsblk at default run level

Comment 6 David 2014-03-19 11:08:01 UTC
Created attachment 876319 [details]
lvmdump at default run level

Comment 7 Peter Rajnoha 2014-03-20 11:25:01 UTC
From the logs:

Mar 19 05:32:49 buttercup pvscan[982]: Request to update PV 15nr2D-jDIM-Rytj-pYJl-Qrhv-tR0z-wCg57t in lvmetad gave response token_mismatch. Reason: token mismatch
Mar 19 05:32:49 buttercup pvscan[1099]: Request to update PV VCCQ01-bD2t-yp6W-vVvr-6bgM-xglo-EUFXd6 in lvmetad gave response token_mismatch. Reason: token mismatch
Mar 19 05:32:51 buttercup pvscan[991]: Request to update PV E2eYRG-lud2-qSjw-LIGA-3sae-Si5f-wkPF51 in lvmetad gave response token_mismatch. Reason: token mismatch

...we've recently added some fixes in this area, mainly https://git.fedorahosted.org/cgit/lvm2.git/commit/?id=1769eddde7dcdd16716d646e14487cc1df6a3efc

Could you please try this build - it includes the latest upstream changes:
  http://koji.fedoraproject.org/koji/taskinfo?taskID=6654431

Comment 8 David 2014-03-23 11:04:15 UTC
Peter, 

I installed the build. Have booted multiple times successfully. 

This seems to have fixed my issue. Thanks!

Comment 9 Peter Rajnoha 2014-03-24 08:38:54 UTC
OK, there should be a new LVM2 upstream release soon - I'll do an official update for F20 then. The package you tested contained all we currently have in upstream and which is prepared for the upcoming LVM2 v2.02.106.

Comment 10 Fedora Update System 2014-04-15 10:29:00 UTC
lvm2-2.02.106-1.fc20 has been submitted as an update for Fedora 20.
https://admin.fedoraproject.org/updates/lvm2-2.02.106-1.fc20

Comment 11 Fedora Update System 2014-04-16 09:25:34 UTC
Package lvm2-2.02.106-1.fc20:
* should fix your issue,
* was pushed to the Fedora 20 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing lvm2-2.02.106-1.fc20'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2014-5241/lvm2-2.02.106-1.fc20
then log in and leave karma (feedback).

Comment 12 Fedora Update System 2014-04-20 01:28:41 UTC
lvm2-2.02.106-1.fc20 has been pushed to the Fedora 20 stable repository.  If problems still persist, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.