Bug 1565187
Summary: | pacemaker can get in a state where a remote is unable to connect to the cluster | ||||||
---|---|---|---|---|---|---|---|
Product: | Red Hat Enterprise Linux 7 | Reporter: | Michele Baldessari <michele> | ||||
Component: | pacemaker | Assignee: | Ken Gaillot <kgaillot> | ||||
Status: | CLOSED ERRATA | QA Contact: | cluster-qe <cluster-qe> | ||||
Severity: | urgent | Docs Contact: | |||||
Priority: | urgent | ||||||
Version: | 7.5 | CC: | abeekhof, aherr, cluster-maint, mkrcmari, mnovacek, nwahl, oblaut, pkomarov, sbradley | ||||
Target Milestone: | rc | Keywords: | Triaged, ZStream | ||||
Target Release: | 7.6 | ||||||
Hardware: | x86_64 | ||||||
OS: | Linux | ||||||
Whiteboard: | |||||||
Fixed In Version: | pacemaker-1.1.18-12.el7 | Doc Type: | Bug Fix | ||||
Doc Text: |
Cause: If an ocf:pacemaker:remote resource has its reconnect_interval option set to a very low value, and the associated remote node needs to be fenced due to failure of the remote resource, Pacemaker could time out the remote resource failure before fencing the remote node.
Consequence: Pacemaker would be unable to reconnect to the remote node.
Fix: Pacemaker now ensures remote resource failure time outs are delayed until any pending fencing completes.
Result: Remote nodes with low reconnect intervals are recovered properly.
|
Story Points: | --- | ||||
Clone Of: | |||||||
: | 1566533 (view as bug list) | Environment: | |||||
Last Closed: | 2018-10-30 07:57:56 UTC | Type: | Bug | ||||
Regression: | --- | Mount Type: | --- | ||||
Documentation: | --- | CRM: | |||||
Verified Versions: | Category: | --- | |||||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
Cloudforms Team: | --- | Target Upstream Version: | |||||
Embargoed: | |||||||
Bug Depends On: | |||||||
Bug Blocks: | 1566533 | ||||||
Attachments: |
|
Description
Michele Baldessari
2018-04-09 15:03:44 UTC
I see: Apr 9 03:53:19 controller-1 crmd[19659]: error: Result of monitor operation for overcloud-novacompute-1 on controller-1: Error Apr 9 03:53:19 controller-1 crmd[19659]: error: remote-node overcloud-novacompute-1 unexpectedly disconneced during monitor operation Apr 9 03:53:19 controller-1 crmd[19659]: notice: State transition S_IDLE -> S_POLICY_ENGINE Apr 9 03:53:19 controller-1 crmd[19659]: notice: Initiating stop operation ip-10.0.0.110_stop_0 on controller-0 I see fencing failing: Apr 9 03:53:19 controller-1 crmd[19659]: notice: Requesting fencing (reboot) of node overcloud-novacompute-1 Apr 9 03:53:19 controller-1 stonith-ng[19655]: notice: Client crmd.19659.946a26a5 wants to fence (reboot) 'overcloud-novacompute-1' with device '(any)' Apr 9 03:53:32 controller-1 stonith-ng[19655]: notice: Operation 'off' [566658] (call 16 from crmd.19659) for host 'overcloud-novacompute-1' with device 'stonith-fence_ipmilan-5254008be2cc' returned: 0 (OK) Apr 9 03:53:32 controller-1 stonith-ng[19655]: notice: Call to stonith-fence_ipmilan-5254008be2cc for 'overcloud-novacompute-1 off' on behalf of crmd.19659@controller-1: OK (0) Apr 9 03:53:42 controller-1 stonith-ng[19655]: notice: Call to stonith-fence_compute-fence-nova for 'overcloud-novacompute-1 off' on behalf of crmd.19659@controller-1: Generic Pacemaker error (-201) Apr 9 03:53:42 controller-1 stonith-ng[19655]: notice: All fencing options to fence overcloud-novacompute-1 for crmd.19659 failed Apr 9 03:53:42 controller-1 stonith-ng[19655]: error: Operation reboot of overcloud-novacompute-1 by controller-2 for crmd.19659: Generic Pacemaker error Apr 9 03:53:42 controller-1 crmd[19659]: notice: Stonith operation 16/27:199:0:69d170fd-52b7-46b4-85cd-cd4a4d08c203: Generic Pacemaker error (-201) Apr 9 03:53:42 controller-1 crmd[19659]: notice: Stonith operation 16 for overcloud-novacompute-1 failed (Generic Pacemaker error): aborting transition. Apr 9 03:53:42 controller-1 crmd[19659]: notice: Transition aborted: Stonith failed which cant be good. That repeats once or twice but then for some reason the cluster forgets the compute is in a failed state This is the "why": [root@controller-1 heat-admin]# crm_diff -n /var/lib/pacemaker/pengine/pe-input-97.bz2 -o /var/lib/pacemaker/pengine/pe-warn-4.bz2 <diff format="2"> <version> <source admin_epoch="0" epoch="104" num_updates="12"/> <target admin_epoch="0" epoch="104" num_updates="17"/> </version> <change operation="delete" path="/cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='overcloud-novacompute-1']/lrm_rsc_op[@id='overcloud-novacompute-1_last_failure_0']"/> <change operation="delete" path="/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-fail-count-overcloud-novacompute-1.monitor_20000']"/> <change operation="delete" path="/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-last-failure-overcloud-novacompute-1.monitor_20000']"/> .... Which was caused by the crmd: Apr 09 07:55:35 [19654] controller-1 cib: info: cib_perform_op: Diff: --- 0.104.14 2 Apr 09 07:55:35 [19654] controller-1 cib: info: cib_perform_op: Diff: +++ 0.104.15 (null) Apr 09 07:55:35 [19654] controller-1 cib: info: cib_perform_op: -- /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='overcloud-novacompute-1']/lrm_rsc_op[@id='overcloud-novacompute-1_last_failure_0'] Apr 09 07:55:35 [19654] controller-1 cib: info: cib_perform_op: + /cib: @num_updates=15 Apr 09 07:55:35 [19654] controller-1 cib: info: cib_process_request: Completed cib_delete operation for section /cib/status/node_state[@uname='controller-1']/lrm/lrm_resources/lrm_resource[@id='overcloud-novacompute-1']/lrm_rsc_op[@id='overcloud-novacompute-1_last_failure_0']: OK (rc=0, origin=controller-1/crmd/2090, version=0.104.15) Apr 09 07:55:35 [19659] controller-1 crmd: info: do_te_invoke: Processing graph 203 (ref=pe_calc-dc-1523260535-572) derived from /var/lib/pacemaker/pengine/pe-warn-4.bz2 Apr 09 07:55:35 [19659] controller-1 crmd: info: te_crm_command: Executing crm-event (26): clear_failcount on controller-1 Which came from this action in pe-warn-4.bz4 <crm_event id="26" operation="clear_failcount" operation_key="overcloud-novacompute-1_clear_failcount_0" on_node="controller-1" on_node_uuid="2"> Pretty clearly that action should depend on a successful stonith operation. Currently it has no prerequisites Created attachment 1419616 [details]
Test case
Ohhh... ./lib/pengine/complex.c-673- /* we want to override any default failure_timeout in use when remote ./lib/pengine/complex.c:674: * reconnect_interval is in use. */ ./lib/pengine/complex.c-675- (*rsc)->failure_timeout = (*rsc)->remote_reconnect_ms / 1000; so this bug actually demonstrates the danger of setting the reconnect interval too low https://github.com/ClusterLabs/pacemaker/commit/4e984bb Requesting z-stream as this is impacting IHA testing Verified , remote-ocf are alive and consistent across reboots and load on a IHA env. tested on: pacemaker-1.1.18-12, OSP13 Test Job results: https://drive.google.com/file/d/1FNQVyfmnDkp6nUUuUnJDA7F4ErcUQSe8/view?usp=sharing Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:3055 |