RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1290140 - deadlock while configuring storage (pvcreate) with looping display (pvs) operations running
Summary: deadlock while configuring storage (pvcreate) with looping display (pvs) oper...
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: lvm2
Version: 7.2
Hardware: x86_64
OS: Linux
unspecified
medium
Target Milestone: rc
: ---
Assignee: David Teigland
QA Contact: cluster-qe@redhat.com
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-12-09 17:57 UTC by Corey Marthaler
Modified: 2021-09-03 12:37 UTC (History)
10 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-08-21 21:19:45 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
log from mckinley-01 (890.99 KB, text/plain)
2015-12-09 21:25 UTC, Corey Marthaler 		no flags Details
log from mckinley-02 (886.30 KB, text/plain)
2015-12-09 21:25 UTC, Corey Marthaler 		no flags Details
log from mckinley-04 (865.78 KB, text/plain)
2015-12-09 21:26 UTC, Corey Marthaler 		no flags Details
View All

Description Corey Marthaler 2015-12-09 17:57:49 UTC 
Description of problem:
I was attempting to reproduce bug 1290129 when my cluster deadlocked.

On all four nodes in the cluster I was running pvs commands every second.

creating lvm devices...
mckinley-01: pvcreate /dev/mapper/mpatha1 /dev/mapper/mpathf1 /dev/mapper/mpathd1 /dev/mapper/mpathe1
[hang]



Dec  9 17:13:21 mckinley-01 crmd[3544]:  notice: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origi]
Dec  9 17:13:21 mckinley-01 pengine[3543]:  notice: Calculated Transition 172: /var/lib/pacemaker/pengine/pe-input-44.bz2
Dec  9 17:13:21 mckinley-01 crmd[3544]:  notice: Transition 172 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lie
Dec  9 17:13:21 mckinley-01 crmd[3544]:  notice: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNA]


Dec  9 17:12:59 mckinley-02 corosync[3001]: [QB    ] conn (3001-3281-30) Nothing in q but got POLLIN on fd:30 (res2:1)
Dec  9 17:12:59 mckinley-02 corosync[3001]: [QB    ] conn (3001-3281-30) Nothing in q but got POLLIN on fd:30 (res2:1)
Dec  9 17:12:59 mckinley-02 corosync[3001]: [QB    ] conn (3001-3281-30) Nothing in q but got POLLIN on fd:30 (res2:1)
Dec  9 17:13:00 mckinley-02 corosync[3001]: [QB    ] conn (3001-3281-30) Nothing in q but got POLLIN on fd:30 (res2:1)
Dec  9 17:13:03 mckinley-02 corosync[3001]: [QB    ] conn (3001-3281-30) Nothing in q but got POLLIN on fd:30 (res2:1)
Dec  9 17:14:03 mckinley-02 corosync[3001]: [QB    ] conn (3001-3281-30) Nothing in q but got POLLIN on fd:30 (res2:1)
Dec  9 17:14:03 mckinley-02 corosync[3001]: [QB    ] conn (3001-3281-30) Nothing in q but got POLLIN on fd:30 (res2:1)
Dec  9 17:15:03 mckinley-02 corosync[3001]: [QB    ] conn (3001-3281-30) Nothing in q but got POLLIN on fd:30 (res2:1)
Dec  9 17:15:03 mckinley-02 corosync[3001]: [QB    ] conn (3001-3281-30) Nothing in q but got POLLIN on fd:30 (res2:1)
Dec  9 17:15:03 mckinley-02 corosync[3001]: [QB    ] conn (3001-3281-30) Nothing in q but got POLLIN on fd:30 (res2:1)
Dec  9 17:16:03 mckinley-02 corosync[3001]: [QB    ] conn (3001-3281-30) Nothing in q but got POLLIN on fd:30 (res2:1)
Dec  9 17:17:03 mckinley-02 corosync[3001]: [QB    ] conn (3001-3281-30) Nothing in q but got POLLIN on fd:30 (res2:1)



  PV                         VG               Fmt  Attr PSize   PFree  PV Tags
  /dev/mapper/mpatha1                              ---       0      0         
  /dev/mapper/mpathb1                              ---       0      0         
  /dev/mapper/mpathc1                              ---       0      0         
  /dev/mapper/mpathd1                              ---       0      0         
  /dev/mapper/mpathe1                              ---       0      0         
  /dev/mapper/mpathf1                              ---       0      0         
  /dev/mapper/mpathg1                              ---       0      0         
  /dev/mapper/mpathh1                              ---       0      0         
  /dev/rhel_mckinley-02/home                       ---       0      0         
  /dev/rhel_mckinley-02/root                       ---       0      0         
  /dev/rhel_mckinley-02/swap                       ---       0      0         
  /dev/sda1                                        ---       0      0         
  /dev/sda2                  rhel_mckinley-02 lvm2 a--  278.97g 60.00m        
  /dev/sdb1                  rhel_mckinley-02 lvm2 a--  279.46g     0         
  Error locking on node 4: Command timed out
  Error locking on node 1: Command timed out
  Error locking on node 3: Command timed out
  Error locking on node 4: Command timed out
  Error locking on node 1: Command timed out
  Error locking on node 3: Command timed out
  Can't get lock for #orphans_pool
  Cannot process volume group #orphans_pool
  Error locking on node 4: Command timed out
  Error locking on node 1: Command timed out
  Error locking on node 3: Command timed out
  Can't get lock for #orphans_lvm2
  Cannot process volume group #orphans_lvm2
  PV                         VG               Fmt  Attr PSize   PFree  PV Tags
  /dev/mapper/mpatha1                              ---       0      0         
  /dev/mapper/mpathb1                              ---       0      0         
  /dev/mapper/mpathc1                              ---       0      0         
  /dev/mapper/mpathd1                              ---       0      0         
  /dev/mapper/mpathe1                              ---       0      0         
  /dev/mapper/mpathf1                              ---       0      0         
  /dev/mapper/mpathg1                              ---       0      0         
  /dev/mapper/mpathh1                              ---       0      0         
  /dev/rhel_mckinley-02/home                       ---       0      0         
  /dev/rhel_mckinley-02/root                       ---       0      0         
  /dev/rhel_mckinley-02/swap                       ---       0      0         
  /dev/sda1                                        ---       0      0         
  /dev/sda2                  rhel_mckinley-02 lvm2 a--  278.97g 60.00m        
  /dev/sdb1                  rhel_mckinley-02 lvm2 a--  279.46g     0         
  Error locking on node 4: Command timed out
  Error locking on node 1: Command timed out
  Error locking on node 3: Command timed out
  Error locking on node 4: Command timed out
  Error locking on node 1: Command timed out
  Error locking on node 3: Command timed out
  Unable to obtain global lock.



## MCKINLEY-01
Dec  9 17:29:43 mckinley-01 kernel: pvcreate        S ffff880fe5fc8ea0     0 48243  48242 0x00000080
Dec  9 17:29:43 mckinley-01 kernel: ffff880036797ac0 0000000000000082 ffff880fdd21f300 ffff880036797fd8
Dec  9 17:29:43 mckinley-01 kernel: ffff880036797fd8 ffff880036797fd8 ffff880fdd21f300 7fffffffffffffff
Dec  9 17:29:43 mckinley-01 kernel: ffff880fdd21f300 0000000000000000 ffff880fe5fc8b40 ffff880fe5fc8ea0
Dec  9 17:29:43 mckinley-01 kernel: Call Trace:
Dec  9 17:29:43 mckinley-01 kernel: [<ffffffff8163a8e9>] schedule+0x29/0x70
Dec  9 17:29:43 mckinley-01 kernel: [<ffffffff816385d9>] schedule_timeout+0x209/0x2d0
Dec  9 17:29:43 mckinley-01 kernel: [<ffffffff810b5d85>] ? check_preempt_curr+0x75/0xa0
Dec  9 17:29:43 mckinley-01 kernel: [<ffffffff810b5dc9>] ? ttwu_do_wakeup+0x19/0xd0
Dec  9 17:29:43 mckinley-01 kernel: [<ffffffff811c3bb8>] ? __kmalloc_node_track_caller+0x58/0x270
Dec  9 17:29:43 mckinley-01 kernel: [<ffffffff810a6806>] ? prepare_to_wait+0x56/0x90
Dec  9 17:29:43 mckinley-01 kernel: [<ffffffff815d274f>] unix_stream_read_generic+0x30f/0x8e0
Dec  9 17:29:43 mckinley-01 kernel: [<ffffffff810a6ae0>] ? wake_up_atomic_t+0x30/0x30
Dec  9 17:29:43 mckinley-01 kernel: [<ffffffff810c34b7>] ? enqueue_entity+0x237/0x890
Dec  9 17:29:43 mckinley-01 kernel: [<ffffffff812888a5>] ? sock_has_perm+0x75/0x90
Dec  9 17:29:43 mckinley-01 kernel: [<ffffffff815d2e14>] unix_stream_recvmsg+0x54/0x70
Dec  9 17:29:43 mckinley-01 kernel: [<ffffffff815cff30>] ? unix_state_double_unlock+0x50/0x50
Dec  9 17:29:43 mckinley-01 kernel: [<ffffffff8150ffb6>] sock_aio_read.part.7+0x146/0x160
Dec  9 17:29:43 mckinley-01 kernel: [<ffffffff810b8c22>] ? default_wake_function+0x12/0x20
Dec  9 17:29:43 mckinley-01 kernel: [<ffffffff8150fff1>] sock_aio_read+0x21/0x30
Dec  9 17:29:43 mckinley-01 kernel: [<ffffffff811ddccd>] do_sync_read+0x8d/0xd0
Dec  9 17:29:43 mckinley-01 kernel: [<ffffffff811de4d5>] vfs_read+0x145/0x170
Dec  9 17:29:43 mckinley-01 kernel: [<ffffffff811def7f>] SyS_read+0x7f/0xe0
Dec  9 17:29:43 mckinley-01 kernel: [<ffffffff816458c9>] system_call_fastpath+0x16/0x1b


Dec  9 17:29:43 mckinley-01 kernel: pvs             S ffff881fe5f49da0     0 48140   3447 0x00000080
Dec  9 17:29:43 mckinley-01 kernel: ffff881fd09b3ac0 0000000000000082 ffff880fe234a280 ffff881fd09b3fd8
Dec  9 17:29:43 mckinley-01 kernel: ffff881fd09b3fd8 ffff881fd09b3fd8 ffff880fe234a280 7fffffffffffffff
Dec  9 17:29:43 mckinley-01 kernel: ffff880fe234a280 0000000000000000 ffff881fe5f49a40 ffff881fe5f49da0
Dec  9 17:29:43 mckinley-01 kernel: Call Trace:
Dec  9 17:29:43 mckinley-01 kernel: [<ffffffff8163a8e9>] schedule+0x29/0x70
Dec  9 17:29:43 mckinley-01 kernel: [<ffffffff816385d9>] schedule_timeout+0x209/0x2d0
Dec  9 17:29:43 mckinley-01 kernel: [<ffffffff810b5d85>] ? check_preempt_curr+0x75/0xa0
Dec  9 17:29:43 mckinley-01 kernel: [<ffffffff810a6806>] ? prepare_to_wait+0x56/0x90
Dec  9 17:29:43 mckinley-01 kernel: [<ffffffff815d274f>] unix_stream_read_generic+0x30f/0x8e0
Dec  9 17:29:43 mckinley-01 kernel: [<ffffffff810a6ae0>] ? wake_up_atomic_t+0x30/0x30
Dec  9 17:29:43 mckinley-01 kernel: [<ffffffff810b0ddf>] ? __wake_up_sync_key+0x4f/0x60
Dec  9 17:29:43 mckinley-01 kernel: [<ffffffff812888a5>] ? sock_has_perm+0x75/0x90
Dec  9 17:29:43 mckinley-01 kernel: [<ffffffff815d2e14>] unix_stream_recvmsg+0x54/0x70
Dec  9 17:29:43 mckinley-01 kernel: [<ffffffff815cff30>] ? unix_state_double_unlock+0x50/0x50
Dec  9 17:29:43 mckinley-01 kernel: [<ffffffff8150ffb6>] sock_aio_read.part.7+0x146/0x160
Dec  9 17:29:43 mckinley-01 kernel: [<ffffffff810b8c22>] ? default_wake_function+0x12/0x20
Dec  9 17:29:43 mckinley-01 kernel: [<ffffffff8150fff1>] sock_aio_read+0x21/0x30
Dec  9 17:29:43 mckinley-01 kernel: [<ffffffff811ddccd>] do_sync_read+0x8d/0xd0
Dec  9 17:29:43 mckinley-01 kernel: [<ffffffff811de4d5>] vfs_read+0x145/0x170
Dec  9 17:29:43 mckinley-01 kernel: [<ffffffff811def7f>] SyS_read+0x7f/0xe0
Dec  9 17:29:43 mckinley-01 kernel: [<ffffffff816458c9>] system_call_fastpath+0x16/0x1b


## MCKINLEY-02
Dec  9 17:30:05 mckinley-02 kernel: corosync        S ffffc9001ca65dc0     0  3002      1 0x00000080
Dec  9 17:30:05 mckinley-02 kernel: ffff881fe752fcd0 0000000000000086 ffff881fe5989700 ffff881fe752ffd8
Dec  9 17:30:05 mckinley-02 kernel: ffff881fe752ffd8 ffff881fe752ffd8 ffff881fe5989700 ffff881fe5989700
Dec  9 17:30:05 mckinley-02 kernel: ffff881fe5989700 0000000000000000 ffff881fe752fde0 ffffc9001ca65dc0
Dec  9 17:30:05 mckinley-02 kernel: Call Trace:
Dec  9 17:30:05 mckinley-02 kernel: [<ffffffff8163a8e9>] schedule+0x29/0x70
Dec  9 17:30:05 mckinley-02 kernel: [<ffffffff810e2464>] futex_wait_queue_me+0xc4/0x120
Dec  9 17:30:05 mckinley-02 kernel: [<ffffffff810e2fd9>] futex_wait+0x179/0x280
Dec  9 17:30:05 mckinley-02 kernel: [<ffffffff810e506e>] do_futex+0xfe/0x5b0
Dec  9 17:30:05 mckinley-02 kernel: [<ffffffff810e55a0>] SyS_futex+0x80/0x180
Dec  9 17:30:05 mckinley-02 kernel: [<ffffffff816458c9>] system_call_fastpath+0x16/0x1b
Dec  9 17:30:05 mckinley-02 kernel: pacemakerd      S ffff880fe61bbbc4     0  3020      1 0x00000080
Dec  9 17:30:05 mckinley-02 kernel: ffff880fe61bba30 0000000000000082 ffff881fe4684500 ffff880fe61bbfd8
Dec  9 17:30:05 mckinley-02 kernel: ffff880fe61bbfd8 ffff880fe61bbfd8 ffff881fe4684500 ffff880fe61bbb88
Dec  9 17:30:05 mckinley-02 kernel: 000000000007a11f 0000000000000000 0000000000000000 ffff880fe61bbbc4
Dec  9 17:30:05 mckinley-02 kernel: Call Trace:
Dec  9 17:30:05 mckinley-02 kernel: [<ffffffff8163a8e9>] schedule+0x29/0x70
Dec  9 17:30:05 mckinley-02 kernel: [<ffffffff81639a02>] schedule_hrtimeout_range_clock+0xb2/0x150
Dec  9 17:30:05 mckinley-02 kernel: [<ffffffff810a95c0>] ? hrtimer_get_res+0x50/0x50
Dec  9 17:30:05 mckinley-02 kernel: [<ffffffff816399f6>] ? schedule_hrtimeout_range_clock+0xa6/0x150
Dec  9 17:30:05 mckinley-02 kernel: [<ffffffff81639ab3>] schedule_hrtimeout_range+0x13/0x20
Dec  9 17:30:05 mckinley-02 kernel: [<ffffffff811f2ba5>] poll_schedule_timeout+0x55/0xb0
Dec  9 17:30:05 mckinley-02 kernel: [<ffffffff811f412d>] do_sys_poll+0x4cd/0x580
Dec  9 17:30:05 mckinley-02 kernel: [<ffffffff81213b38>] ? generic_write_end+0x78/0xb0
Dec  9 17:30:05 mckinley-02 kernel: [<ffffffffa03a47b9>] ? xfs_vm_write_end+0x29/0x80 [xfs]
Dec  9 17:30:05 mckinley-02 kernel: [<ffffffff81169e06>] ? generic_file_buffered_write+0x1b6/0x290
Dec  9 17:30:05 mckinley-02 kernel: [<ffffffff811f2ac0>] ? poll_select_copy_remaining+0x150/0x150
Dec  9 17:30:05 mckinley-02 kernel: [<ffffffff811f2ac0>] ? poll_select_copy_remaining+0x150/0x150
Dec  9 17:30:05 mckinley-02 kernel: [<ffffffff811f2ac0>] ? poll_select_copy_remaining+0x150/0x150
Dec  9 17:30:05 mckinley-02 kernel: [<ffffffff811f2ac0>] ? poll_select_copy_remaining+0x150/0x150
Dec  9 17:30:05 mckinley-02 kernel: [<ffffffff811f2ac0>] ? poll_select_copy_remaining+0x150/0x150
Dec  9 17:30:05 mckinley-02 kernel: [<ffffffff811ddd9d>] ? do_sync_write+0x8d/0xd0
Dec  9 17:30:05 mckinley-02 kernel: [<ffffffff8128bc6e>] ? file_has_perm+0xae/0xc0
Dec  9 17:30:05 mckinley-02 kernel: [<ffffffff8122bb27>] ? eventfd_ctx_read+0x67/0x210
Dec  9 17:30:05 mckinley-02 kernel: [<ffffffff810b8c10>] ? wake_up_state+0x20/0x20
Dec  9 17:30:05 mckinley-02 kernel: [<ffffffff8101c829>] ? read_tsc+0x9/0x10
Dec  9 17:30:05 mckinley-02 kernel: [<ffffffff810d814c>] ? ktime_get_ts64+0x4c/0xf0
Dec  9 17:30:05 mckinley-02 kernel: [<ffffffff811f42e4>] SyS_poll+0x74/0x110
Dec  9 17:30:05 mckinley-02 kernel: [<ffffffff816458c9>] system_call_fastpath+0x16/0x1b



Version-Release number of selected component (if applicable):
3.10.0-325.el7.x86_64
corosync-2.3.4-7.el7.x86_64   BUILT: Mon 22 Jun 2015 09:32:50 AM CDT
pacemaker-1.1.13-10.el7.x86_64   BUILT: Thu 08 Oct 2015 06:06:55 PM CDT

lvm2-2.02.130-5.el7    BUILT: Wed Oct 14 08:27:29 CDT 2015
lvm2-libs-2.02.130-5.el7    BUILT: Wed Oct 14 08:27:29 CDT 2015
lvm2-cluster-2.02.130-5.el7    BUILT: Wed Oct 14 08:27:29 CDT 2015
device-mapper-1.02.107-5.el7    BUILT: Wed Oct 14 08:27:29 CDT 2015
device-mapper-libs-1.02.107-5.el7    BUILT: Wed Oct 14 08:27:29 CDT 2015
device-mapper-event-1.02.107-5.el7    BUILT: Wed Oct 14 08:27:29 CDT 2015
device-mapper-event-libs-1.02.107-5.el7    BUILT: Wed Oct 14 08:27:29 CDT 2015
device-mapper-persistent-data-0.5.5-1.el7    BUILT: Thu Aug 13 09:58:10 CDT 2015
cmirror-2.02.130-5.el7    BUILT: Wed Oct 14 08:27:29 CDT 2015
Comment 1 Corey Marthaler 2015-12-09 21:25:12 UTC 
Created attachment 1104138 [details]
log from mckinley-01
Comment 2 Corey Marthaler 2015-12-09 21:25:49 UTC 
Created attachment 1104139 [details]
log from mckinley-02
Comment 3 Corey Marthaler 2015-12-09 21:26:20 UTC 
Created attachment 1104140 [details]
log from mckinley-04
Comment 5 Jonathan Earl Brassow 2016-06-30 16:33:49 UTC 
Possible duplicate of Bug 1194667 ?
Comment 6 Jonathan Earl Brassow 2017-07-27 20:37:01 UTC 
(In reply to Jonathan Earl Brassow from comment #5)
> Possible duplicate of Bug 1194667 ?

and possibly bug 1263356 ?
Note You need to log in before you can comment on or make changes to this bug.