1638084 – upconvert attempt after secondary mirror leg failure causes clvmd to hang and gfs to withdraw [rhel-7.6.z]

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1638084 - upconvert attempt after secondary mirror leg failure causes clvmd to hang and gfs to withdraw [rhel-7.6.z]

Summary: upconvert attempt after secondary mirror leg failure causes clvmd to hang and...

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Enterprise Linux 7
Classification:	Red Hat
Component:	lvm2
Sub Component:
Version:	7.6
Hardware:	x86_64
OS:	Linux
Priority:	high
Severity:	high
Target Milestone:	rc
Target Release:	---
Assignee:	LVM and device-mapper development team
QA Contact:	cluster-qe@redhat.com
Docs Contact:
URL:
Whiteboard:
Depends On:	1628529
Blocks:
TreeView+	depends on / blocked

Reported:	2018-10-10 15:46 UTC by Oneata Mircea Teodor
Modified:	2021-09-03 12:55 UTC (History)
CC List:	17 users (show)
Fixed In Version:	lvm2-2.02.180-9.el7.1
Doc Type:	If docs needed, set a value
Doc Text:	Previously, clustered LVM caused the GFS file system to withdraw when the secondary mirror leg failed with auto allocation. As a consequence, all commands using the lvm utility became unresponsive. With this update, the problem has been fixed.
Clone Of:	1628529
Environment:
Last Closed:	2018-10-30 12:22:03 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Product Errata	RHBA-2018:3344	0	None	None	None	2018-10-30 12:22:11 UTC

Description Oneata Mircea Teodor 2018-10-10 15:46:50 UTC

This bug has been copied from bug #1628529 and has been proposed to be backported to 7.6 z-stream (EUS).

Comment 4 Corey Marthaler 2018-10-11 21:59:38 UTC

This test case passes with the latest lvm 7.6.z build (the rpm version listed in the errata): lvm2-2.02.180-10.el7_6.1    BUILT: Wed Oct 10 12:43:42 CDT 2018


3.10.0-957.el7.x86_64

lvm2-2.02.180-10.el7_6.1    BUILT: Wed Oct 10 12:43:42 CDT 2018
lvm2-libs-2.02.180-10.el7_6.1    BUILT: Wed Oct 10 12:43:42 CDT 2018
lvm2-cluster-2.02.180-10.el7_6.1    BUILT: Wed Oct 10 12:43:42 CDT 2018
lvm2-lockd-2.02.180-10.el7_6.1    BUILT: Wed Oct 10 12:43:42 CDT 2018
lvm2-python-boom-0.9-11.el7    BUILT: Mon Sep 10 04:49:22 CDT 2018
cmirror-2.02.180-10.el7_6.1    BUILT: Wed Oct 10 12:43:42 CDT 2018
device-mapper-1.02.149-10.el7_6.1    BUILT: Wed Oct 10 12:43:42 CDT 2018
device-mapper-libs-1.02.149-10.el7_6.1    BUILT: Wed Oct 10 12:43:42 CDT 2018
device-mapper-event-1.02.149-10.el7_6.1    BUILT: Wed Oct 10 12:43:42 CDT 2018
device-mapper-event-libs-1.02.149-10.el7_6.1    BUILT: Wed Oct 10 12:43:42 CDT 2018
device-mapper-persistent-data-0.7.3-3.el7    BUILT: Tue Nov 14 05:07:18 CST 2017


Enabling mirror allocate fault policies on: mckinley-01 mckinley-02 mckinley-03
================================================================================
Iteration 0.1 started at Thu Oct 11 16:31:55 CDT 2018
================================================================================
Scenario kill_secondary_synced_3_legs: Kill secondary leg of synced 3 leg mirror(s)
********* Mirror hash info for this scenario *********
* names:              syncd_secondary_3legs_1
* sync:               1
* striped:            0
* leg devices:        /dev/mapper/mpathc1 /dev/mapper/mpatha1 /dev/mapper/mpathe1
* log devices:        /dev/mapper/mpathd1
* no MDA devices:     
* failpv(s):          /dev/mapper/mpatha1
* failnode(s):        mckinley-01 mckinley-02 mckinley-03
* lvmetad:            0
* leg fault policy:   allocate
* log fault policy:   allocate
******************************************************

Creating mirror(s) on mckinley-03...
mckinley-03: lvcreate -ay --type mirror -m 2 -n syncd_secondary_3legs_1 -L 500M helter_skelter /dev/mapper/mpathc1:0-2400 /dev/mapper/mpatha1:0-2400 /dev/mapper/mpathe1:0-2400 /dev/mapper/mpathd1:0-150

Current mirror/raid device structure(s):
  LV                                 Attr       LSize   Cpy%Sync Devices
   syncd_secondary_3legs_1            mwi-a-m--- 500.00m 8.80     syncd_secondary_3legs_1_mimage_0(0),syncd_secondary_3legs_1_mimage_1(0),syncd_secondary_3legs_1_mimage_2(0)
   [syncd_secondary_3legs_1_mimage_0] Iwi-aom--- 500.00m          /dev/mapper/mpathc1(0)
   [syncd_secondary_3legs_1_mimage_1] Iwi-aom--- 500.00m          /dev/mapper/mpatha1(0)
   [syncd_secondary_3legs_1_mimage_2] Iwi-aom--- 500.00m          /dev/mapper/mpathe1(0)
   [syncd_secondary_3legs_1_mlog]     lwi-aom---   4.00m          /dev/mapper/mpathd1(0)

Waiting until all mirror|raid volumes become fully syncd...
   1/1 mirror(s) are fully synced: ( 100.00% )
Sleeping 15 sec
Sleeping 15 sec
Sleeping 15 sec

Creating gfs2 on top of mirror(s) on mckinley-01...
mkfs.gfs2 -o align=0 -J 32M -j 3 -p lock_dlm -t MCKINLEY:gfs1 /dev/helter_skelter/syncd_secondary_3legs_1 -O
Mounting mirrored gfs2 filesystems on mckinley-01...
Mounting mirrored gfs2 filesystems on mckinley-02...
Mounting mirrored gfs2 filesystems on mckinley-03...

PV=/dev/mapper/mpatha1
        syncd_secondary_3legs_1_mimage_1: 4.1
PV=/dev/mapper/mpatha1
        syncd_secondary_3legs_1_mimage_1: 4.1

Writing verification files (checkit) to mirror(s) on...
        ---- mckinley-01 ----
        ---- mckinley-02 ----
        ---- mckinley-03 ----

Sleeping 15 seconds to get some outsanding I/O locks before the failure 
Verifying files (checkit) on mirror(s) on...
        ---- mckinley-01 ----
        ---- mckinley-02 ----
        ---- mckinley-03 ----

Failing path sdb on mckinley-01
Failing path sdj on mckinley-01
Failing path sdr on mckinley-01
Failing path sdz on mckinley-01
Failing path sdc on mckinley-02
Failing path sdk on mckinley-02
Failing path sds on mckinley-02
Failing path sdaa on mckinley-02
Failing path sdc on mckinley-03
Failing path sdk on mckinley-03
Failing path sds on mckinley-03
Failing path sdaa on mckinley-03

Attempting I/O to cause mirror down conversion(s) on mckinley-01
dd if=/dev/zero of=/mnt/syncd_secondary_3legs_1/ddfile count=10 bs=4M
10+0 records in
10+0 records out
41943040 bytes (42 MB) copied, 0.189207 s, 222 MB/s
Verifying current sanity of lvm after the failure

Current mirror/raid device structure(s):
  Couldn't find device with uuid CaiqAq-BQm4-dKYy-GcZO-N9w1-l0Bc-X8Tf0I.
  LV                                    Attr       LSize   Cpy%Sync Devices
   syncd_secondary_3legs_1               cwi-aom--- 500.00m 100.00   syncd_secondary_3legs_1_mimagetmp_3(0),syncd_secondary_3legs_1_mimage_3(0)
   [syncd_secondary_3legs_1_mimage_0]    iwi-aom--- 500.00m          /dev/mapper/mpathc1(0)
   [syncd_secondary_3legs_1_mimage_2]    iwi-aom--- 500.00m          /dev/mapper/mpathe1(0)
   [syncd_secondary_3legs_1_mimage_3]    iwi-aom--- 500.00m          /dev/mapper/mpathb1(0)
   [syncd_secondary_3legs_1_mimagetmp_3] mwi-aom--- 500.00m 100.00   syncd_secondary_3legs_1_mimage_0(0),syncd_secondary_3legs_1_mimage_2(0)   
   [syncd_secondary_3legs_1_mlog]        lwi-aom---   4.00m          /dev/mapper/mpathd1(0)

Verifying FAILED device /dev/mapper/mpatha1 is *NOT* in the volume(s)
Verifying LOG device(s) /dev/mapper/mpathd1 *ARE* in the mirror(s)
Verifying LEG device /dev/mapper/mpathc1 *IS* in the volume(s)
Verifying LEG device /dev/mapper/mpathe1 *IS* in the volume(s)
verify the dm devices associated with /dev/mapper/mpatha1 have been removed as expected
Checking REMOVAL of syncd_secondary_3legs_1_mimage_1 on:  mckinley-01 mckinley-02 mckinley-03
verify the newly allocated dm devices were added as a result of the failures
Checking EXISTENCE of syncd_secondary_3legs_1_mimage_3 on:  mckinley-01 mckinley-02 mckinley-03

Verify that the mirror image order remains the same after the down conversion
EXPECTED LEG ORDER: /dev/mapper/mpathc1 /dev/mapper/mpathe1 unknown
ACTUAL LEG ORDER: /dev/mapper/mpathc1 /dev/mapper/mpathe1 /dev/mapper/mpathb1
Verifying files (checkit) on mirror(s) on...
        ---- mckinley-01 ----
        ---- mckinley-02 ----
        ---- mckinley-03 ----

Enabling path sdb on mckinley-01
Enabling path sdj on mckinley-01
Enabling path sdr on mckinley-01
Enabling path sdz on mckinley-01
Enabling path sdc on mckinley-02
Enabling path sdk on mckinley-02
Enabling path sds on mckinley-02
Enabling path sdaa on mckinley-02
Enabling path sdc on mckinley-03
Enabling path sdk on mckinley-03
Enabling path sds on mckinley-03
Enabling path sdaa on mckinley-03

-------------------------------------------------------------------------------
Force a vgreduce to either clean up the corrupt additional LV or lack of PV MDA
-------------------------------------------------------------------------------
  Scan of VG helter_skelter from /dev/mapper/mpatha1 found metadata seqno 15 vs previous 23.
  Scan of VG helter_skelter from /dev/mapper/mpatha1 found metadata seqno 15 vs previous 23.
  WARNING: Inconsistent metadata found for VG helter_skelter - updating to use version 23
Recreating PVs /dev/mapper/mpatha1 and then extending back into helter_skelter
mckinley-03 pvcreate /dev/mapper/mpatha1
  Scan of VG helter_skelter from /dev/mapper/mpatha1 found metadata seqno 24 vs previous 25.
  Scan of VG helter_skelter from /dev/mapper/mpatha1 found metadata seqno 24 vs previous 25.
  Scan of VG helter_skelter from /dev/mapper/mpatha1 found metadata seqno 24 vs previous 25.
  WARNING: Inconsistent metadata found for VG helter_skelter - updating to use version 25
  WARNING: Removing PV /dev/mapper/mpatha1 (CaiqAq-BQm4-dKYy-GcZO-N9w1-l0Bc-X8Tf0I) that no longer belongs to VG helter_skelter
mckinley-03 vgextend helter_skelter /dev/mapper/mpatha1

Verify that each of the raid repairs finished successfully
Checking for leftover '-missing_0_0' or 'unknown devices'

Checking for PVs marked as missing (a-m)...
Checking for PVs marked as missing (a-m)...
Checking for PVs marked as missing (a-m)...
Waiting until all mirror|raid volumes become fully syncd...
   1/1 mirror(s) are fully synced: ( 100.00% )
Sleeping 15 sec

Verifying files (checkit) on mirror(s) on...
        ---- mckinley-01 ----
        ---- mckinley-02 ----
        ---- mckinley-03 ----

Stopping the io load (collie/xdoio) on mirror(s)
Unmounting gfs and removing mnt point on mckinley-01...
Unmounting gfs and removing mnt point on mckinley-02...
Unmounting gfs and removing mnt point on mckinley-03...

Deactivating and removing mirror(s)
Checking for left over mirror dm devices on
         ...mckinley-01 ...mckinley-02 ...mckinley-03

Comment 10 errata-xmlrpc 2018-10-30 12:22:03 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:3344

Note You need to log in before you can comment on or make changes to this bug.