Bug 1069808

Summary: "Request to update PV ... in lvmetad gave response token_mismatch" during net-install boot
Product: Red Hat Enterprise Linux 7 Reporter: Marian Csontos <mcsontos>
Component: lvm2Assignee: Petr Rockai <prockai>
lvm2 sub component: LVM Metadata / lvmetad QA Contact: Cluster QE <mspqa-list>
Status: CLOSED CURRENTRELEASE Docs Contact:
Severity: unspecified    
Priority: unspecified CC: agk, heinzm, jbrassow, msnitzer, prajnoha, prockai, zkabelac
Version: 7.0   
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-02-28 13:39:58 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
journal after the installation is over
none
lvmdump none

Description Marian Csontos 2014-02-25 17:00:10 UTC
Description of problem:

During testing more complex LVM layouts I am seeing messages like these on console:

    Feb 25 15:39:39 localhost pvscan[1138]: Request to update PV WC8isV-W2Fl-XluT-9YdW-OdOv-bmv1-jRJJg7 in lvmetad gave response token_mismatch. Reason: token        mismatch
    Feb 25 15:39:39 localhost pvscan[1138]: Update of lvmetad failed. This is a serious problem.
    Feb 25 15:39:39 localhost pvscan[1138]: It is strongly recommended that you restart lvmetad immediately.
    Feb 25 15:39:39 localhost systemd[1]: lvm2-pvscan@252:129.service: main process exited, code=exited, status=5/NOTINSSTALLED
    Feb 25 15:39:39 localhost systemd[1]: Failed to start LVM2 PV scan on device 252:129.
    Feb 25 15:39:39 localhost systemd[1]: Unit lvm2-pvscan@252:129.service entered failed state.

I have 8 PVs with MDAs and this happens for 6 of them, remaining 2 usually run fine.

This does not happen after the system is installed.

Version-Release number of selected component (if applicable):
anaconda 19.31.60-1
lvm2-2.02.105-4.el7
compose RHEL-7.0-20140220.2

How reproducible:
100% (with given layout)

Steps to Reproduce:
1. boot net-boot image in VM with given layout

Actual results:

Scary messages about lvmetad asking for immediate restart of the service:

    Feb 25 15:39:39 localhost pvscan[1138]: Update of lvmetad failed. This is a serious problem.
    Feb 25 15:39:39 localhost pvscan[1138]: It is strongly recommended that you restart lvmetad immediately.

Expected results:

Clean boot.

Additional info:

Comment 2 Marian Csontos 2014-02-25 17:05:15 UTC
This may be related to udev rules in boot image being different form the real thing.

Also default lvm.conf is used.

In journal it looks like LVs were activated before all PVs were scanned but that's likely just a timing issue with multiple processes output being gathered in different order.

Comment 3 Marian Csontos 2014-02-26 07:38:20 UTC
Created attachment 867824 [details]
journal after the installation is over

Comment 4 Peter Rajnoha 2014-02-26 12:26:19 UTC
Just FYI: the "token mismatch" error is also seen in recent bug #1069764 (F20). So the question is - how do we reach the situation in which "token mismatch" is returned? What are all the possibilities to get there?

Comment 5 Peter Rajnoha 2014-02-26 12:28:06 UTC
What is the LVM layout used here exactly? (what are all the PV UUIDs used? So we can match them against the journal log...)

Comment 6 Marian Csontos 2014-02-27 08:51:31 UTC
No longer reproducible with lvm2-2.02.105-7.el7.

Comment 7 Marian Csontos 2014-02-27 11:25:59 UTC
Created attachment 868470 [details]
lvmdump

If anyone wants to dive deeper here's lvmdump.

Comment 8 Petr Rockai 2014-02-28 10:52:43 UTC
Should have been fixed in 1769eddde7dcdd16716d646e14487cc1df6a3efc so it's good that it is no longer reproducible.