Bug 596537 - Unable to determine mirror sync status of mirror causes conversion to fail
Unable to determine mirror sync status of mirror causes conversion to fail
Status: CLOSED NOTABUG
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: lvm2 (Show other bugs)
6.0
All Linux
medium Severity medium
: rc
: ---
Assigned To: Jonathan Earl Brassow
Corey Marthaler
: Reopened
Depends On: 591710
Blocks:
  Show dependency treegraph
 
Reported: 2010-05-26 17:44 EDT by Corey Marthaler
Modified: 2010-09-28 23:36 EDT (History)
10 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
The sync state of an inactive LVM mirror cannot be determined. Consequently, the primary device of an LVM mirror can only be removed when the mirror is in-sync.
Story Points: ---
Clone Of: 591710
Environment:
Last Closed: 2010-08-11 16:03:11 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Corey Marthaler 2010-05-26 17:44:48 EDT
This issue exists in 6.0 as well.

lvm2-2.02.65-1.el6    BUILT: Tue May 18 04:46:06 CDT 2010
lvm2-libs-2.02.65-1.el6    BUILT: Tue May 18 04:46:06 CDT 2010
lvm2-cluster-2.02.65-1.el6    BUILT: Tue May 18 04:46:06 CDT 2010
device-mapper-1.02.48-1.el6    BUILT: Tue May 18 04:46:06 CDT 2010
device-mapper-libs-1.02.48-1.el6    BUILT: Tue May 18 04:46:06 CDT 2010
device-mapper-event-1.02.48-1.el6    BUILT: Tue May 18 04:46:06 CDT 2010
device-mapper-event-libs-1.02.48-1.el6    BUILT: Tue May 18 04:46:06 CDT 2010
cmirror-2.02.65-1.el6    BUILT: Wed May 19 11:19:57 CDT 2010


+++ This bug was initially created as a clone of Bug #591710 +++

Description of problem:

Test output:
---------------------------------------------
 base legs 3; convert to 1 legs;
 active 0; sync=0; log=core; pvs=sufficient
---------------------------------------------
creating a base mirror on grant-02
deactivating base volume before convert on grant-01

Converting from 3 leg(s) to 1 leg(s) on grant-02
lvconvert --mirrorlog core -m 1 centipede/centi_base /dev/sdc1 /dev/sdc3
  Unable to determine mirror sync status of centipede/centi_base.
  Unable to remove primary mirror image while mirror is not in-sync

Version-Release number of selected component (if applicable):
2.6.18-192.el5

lvm2-2.02.56-8.el5_5.3    BUILT: Wed Apr 28 03:08:59 CDT 2010
lvm2-cluster-2.02.56-7.el5_5.1    BUILT: Thu Apr 22 07:02:04 CDT 2010
device-mapper-1.02.39-1.el5_5.2    BUILT: Thu Apr 22 04:30:57 CDT 2010
cmirror-1.1.39-8.el5    BUILT: Wed Mar  3 09:31:58 CST 2010
kmod-cmirror-0.1.22-3.el5    BUILT: Tue Dec 22 13:39:47 CST 2009


How reproducible:
Often

--- Additional comment from cmarthal@redhat.com on 2010-05-12 17:06:37 EDT ---

Attaching full -vvvv log...

#format_text/archiver.c:127     Archiving volume group "centipede" metadata (seqno 134).
#lvconvert.c:683     Setting logging type to core
#metadata/mirror.c:533       Reducing mirror set from 4 to 2 image(s) and no log volume.
#activate/activate.c:456         Getting device info for centipede-centi_base
#ioctl/libdm-iface.c:1740         dm info  LVM-nyIxipVlialS1ACBMc3hh7CjLNiaFtwIfGMOXriy0aQwURD6m5RYES3GpkfGQcgp NF   [16384]
#ioctl/libdm-iface.c:1740         dm info  nyIxipVlialS1ACBMc3hh7CjLNiaFtwIfGMOXriy0aQwURD6m5RYES3GpkfGQcgp NF   [16384]
#ioctl/libdm-iface.c:1740         dm info centipede-centi_base  NF   [16384]
#metadata/mirror.c:478   Unable to determine mirror sync status of centipede/centi_base.
#metadata/mirror.c:557   Unable to remove primary mirror image while mirror is not in-sync
#metadata/mirror.c:558         <backtrace>
#metadata/mirror.c:726         <backtrace>
#lvconvert.c:732         <backtrace>
#lvconvert.c:1022         <backtrace>
#locking/cluster_locking.c:466       Locking VG V_centipede UN (VG) (0x6

--- Additional comment from cmarthal@redhat.com on 2010-05-12 17:07:50 EDT ---

Created an attachment (id=413564)
lvconvert -vvvv --mirrorlog core -m 1 centipede/centi_base /dev/sdc1 /dev/sdc3 > /tmp/lvconvert 2>&1

--- Additional comment from cmarthal@redhat.com on 2010-05-12 17:46:28 EDT ---

No dm stuff shows up before or after the failure due to it being inactive:

Converting from 3 leg(s) to 0 leg(s) on grant-02

info: 
Name                Maj Min Stat Open Targ Event  UUID
 VolGroup00-LogVol01 253   1 L--w    1    1      0 LVM-RDJ1P8ieOwNeioALQCGSomXR23jOJrs4qApUeTmnKm99geaXhXNadZzcaEpTNHfp
 VolGroup00-LogVol00 253   0 L--w    1    1      0 LVM-RDJ1P8ieOwNeioALQCGSomXR23jOJrs4eKKvOyJhYxYQVza81JizSt4XjAl1e6bT

table: 
 VolGroup00-LogVol01: 0 20578304 linear 8:2 135397760
 VolGroup00-LogVol00: 0 135397376 linear 8:2 384

status: 
 VolGroup00-LogVol01: 0 20578304 linear
 VolGroup00-LogVol00: 0 135397376 linear

lvconvert --mirrorlog core -m 0 centipede/centi_base /dev/sdc3 /dev/sdc2 /dev/sdc1
  Unable to determine mirror sync status of centipede/centi_base.
  Unable to remove primary mirror image while mirror is not in-sync
couldn't convert on grant-02

info: 
Name                Maj Min Stat Open Targ Event  UUID
 VolGroup00-LogVol01 253   1 L--w    1    1      0 LVM-RDJ1P8ieOwNeioALQCGSomXR23jOJrs4qApUeTmnKm99geaXhXNadZzcaEpTNHfp
 VolGroup00-LogVol00 253   0 L--w    1    1      0 LVM-RDJ1P8ieOwNeioALQCGSomXR23jOJrs4eKKvOyJhYxYQVza81JizSt4XjAl1e6bT

table: 
 VolGroup00-LogVol01: 0 20578304 linear 8:2 135397760
 VolGroup00-LogVol00: 0 135397376 linear 8:2 384

status: 
 VolGroup00-LogVol01: 0 20578304 linear
 VolGroup00-LogVol00: 0 135397376 linear

--- Additional comment from cmarthal@redhat.com on 2010-05-12 18:48:13 EDT ---

It appears that this bug only occurs when down converting, and when specifying the pvs in the mirror to remove. 

Fails:
Converting from 3 leg(s) to 1 leg(s) on grant-02
lvconvert --mirrorlog core -m 1 centipede/centi_base /dev/sdc1 /dev/sdc3

Passes:
Converting from 3 leg(s) to 1 leg(s) on grant-02
lvconvert --mirrorlog core -m 1 centipede/centi_base

I'll need further testing to verify if this is always true however.

--- Additional comment from cmarthal@redhat.com on 2010-05-13 14:05:26 EDT ---

Further testing passed when not specifying the pvs to remove when down converting.
Comment 1 RHEL Product and Program Management 2010-05-26 17:46:24 EDT
This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux major release.  Product Management has requested further
review of this request by Red Hat Engineering, for potential inclusion in a Red
Hat Enterprise Linux Major release.  This request is not yet committed for
inclusion.
Comment 3 Jonathan Earl Brassow 2010-07-20 12:15:47 EDT
corey, can you try again.  This works for me.
Comment 6 Corey Marthaler 2010-07-27 16:07:00 EDT
This is no longer reproducible with the latest code:
 2.6.32-44.el6.x86_64

lvm2-2.02.69-2.el6    BUILT: Fri Jul  2 07:26:01 CDT 2010
lvm2-libs-2.02.69-2.el6    BUILT: Fri Jul  2 07:26:01 CDT 2010
lvm2-cluster-2.02.69-2.el6    BUILT: Fri Jul  2 07:26:01 CDT 2010
device-mapper-1.02.51-2.el6    BUILT: Fri Jul  2 07:26:01 CDT 2010
device-mapper-libs-1.02.51-2.el6    BUILT: Fri Jul  2 07:26:01 CDT 2010
device-mapper-event-1.02.51-2.el6    BUILT: Fri Jul  2 07:26:01 CDT 2010
device-mapper-event-libs-1.02.51-2.el6    BUILT: Fri Jul  2 07:26:01 CDT 2010
cmirror-2.02.69-2.el6    BUILT: Fri Jul  2 07:26:01 CDT 2010


root@grant-01 ~]# lvcreate -m 3 -n centi_base --mirrorlog core -L 100M centipede
  Logical volume "centi_base" created

[root@grant-01 ~]# lvs -a -o +devices
  LV                    VG         Attr   LSize   Log Copy%  Devices
  centi_base            centipede  mwi-a- 100.00m     84.00  centi_base_mimage_0(0),centi_base_mimage_1(0),centi_base_mimage_2(0),centi_base_mimage_3(0)
  [centi_base_mimage_0] centipede  Iwi-ao 100.00m            /dev/sdc6(0)
  [centi_base_mimage_1] centipede  Iwi-ao 100.00m            /dev/sdc5(0)
  [centi_base_mimage_2] centipede  Iwi-ao 100.00m            /dev/sdc3(0)
  [centi_base_mimage_3] centipede  Iwi-ao 100.00m            /dev/sdc2(0)

[root@grant-01 ~]# lvchange -an centipede

[root@grant-01 ~]# lvconvert --mirrorlog core -m 1 centipede/centi_base /dev/sdc2 /dev/sdc3
  Logical volume centi_base converted.
Comment 7 Corey Marthaler 2010-07-27 17:02:41 EDT
Check that! I let the tests continue on, and apparently this is a timing issue because the test eventually hit this bug again.

[root@grant-01 ~]# lvs -a -o +devices
  LV                    VG         Attr   LSize   Log Copy%  Convert Devices
  centi_base            centipede  mwi--- 700.00m                     centi_base_mimage_0(0),centi_base_mimage_1(0),centi_base_mimage_2(0),centi_base_mimage_3(0)
  [centi_base_mimage_0] centipede  Iwi--- 700.00m                     /dev/sdc6(0)
  [centi_base_mimage_1] centipede  Iwi--- 700.00m                     /dev/sdc5(0)
  [centi_base_mimage_2] centipede  Iwi--- 700.00m                     /dev/sdc3(0)
  [centi_base_mimage_3] centipede  Iwi--- 700.00m                     /dev/sdc2(0)

[root@grant-01 ~]# lvconvert --mirrorlog core -m 1 centipede/centi_base /dev/sdc5 /dev/sdc6
  Unable to determine mirror sync status of centipede/centi_base.
  Unable to remove primary mirror image while mirror is not in-sync

2.6.32-44.el6.x86_64

lvm2-2.02.69-2.el6    BUILT: Fri Jul  2 07:26:01 CDT 2010
lvm2-libs-2.02.69-2.el6    BUILT: Fri Jul  2 07:26:01 CDT 2010
lvm2-cluster-2.02.69-2.el6    BUILT: Fri Jul  2 07:26:01 CDT 2010
device-mapper-1.02.51-2.el6    BUILT: Fri Jul  2 07:26:01 CDT 2010
device-mapper-libs-1.02.51-2.el6    BUILT: Fri Jul  2 07:26:01 CDT 2010
device-mapper-event-1.02.51-2.el6    BUILT: Fri Jul  2 07:26:01 CDT 2010
device-mapper-event-libs-1.02.51-2.el6    BUILT: Fri Jul  2 07:26:01 CDT 2010
cmirror-2.02.69-2.el6    BUILT: Fri Jul  2 07:26:01 CDT 2010
Comment 11 Jonathan Earl Brassow 2010-08-11 16:03:11 EDT
I didn't pick up on this right away, but this is expected behavior.

When the sync status of a mirror cannot be determine (like when the mirror is
not active), the primary device cannot be removed.  This is because the other
legs of the mirror may be - to varying degrees - out-of-sync WRT the primary.

It you look at the commands that illustrate the issue and the commands that
succeed, the difference is that you are specifying the primary device on the
command line in the instances that fail.  This is also why the issue cannot be
hit when not specifying devices on the command line - because in that case, the
primary will never be chosen.
Comment 13 Corey Marthaler 2010-08-11 17:43:38 EDT
I'll update the tests to either not remove the primary when not insync, or to accept the warning message.
Comment 14 Jonathan Earl Brassow 2010-08-29 15:23:45 EDT
    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
The primary device of an LVM mirror cannot be removed unless the mirror is in-sync.  This is because the validity of the remaining devices is unknown in this case.  When an LVM mirror is inactive, its sync state cannot be determined.  Thus, the primary image of an LVM mirror cannot be removed while the mirror is inactive.
Comment 16 Ryan Lerch 2010-09-28 23:36:09 EDT
    Technical note updated. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    Diffed Contents:
@@ -1 +1 @@
-The primary device of an LVM mirror cannot be removed unless the mirror is in-sync.  This is because the validity of the remaining devices is unknown in this case.  When an LVM mirror is inactive, its sync state cannot be determined.  Thus, the primary image of an LVM mirror cannot be removed while the mirror is inactive.+The sync state of an inactive LVM mirror cannot be determined. Consequently, the primary device of an LVM mirror can only be removed when the mirror is in-sync.

Note You need to log in before you can comment on or make changes to this bug.