Bug 1364553
| Summary: | failed 'reboot' action with paired switched PDUs leaves no power to the node | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 6 | Reporter: | Madison Kelly <mkelly> |
| Component: | cluster | Assignee: | Marek Grac <mgrac> |
| Status: | CLOSED NOTABUG | QA Contact: | cluster-qe <cluster-qe> |
| Severity: | low | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 6.8 | CC: | abeekhof, ccaulfie, cluster-maint, mkelly, rpeterso, teigland |
| Target Milestone: | rc | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2016-09-06 08:34:38 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
Madison Kelly
2016-08-05 17:01:35 UTC
Hi, I'm not sure if I understand it correctly, so situation is: * node is fenced by IPMI and if IPMI fails then PDU fencing is used * scenario: 1) node is ON, ipmi responds ON, fence-PDU plug ON 2) unplug node from PDU ==> node is ON, IPMI is working, fence-PDU is working but its state is independent of node 3) cut power to node ==> node is OFF, IPMI (fails/connection timeout), fence-PDU reboot plug 4) plug PDU back ==> node is OFF, IPMI is failing, fence-PDU is able to connect In pacemaker cluster, we will notice that the fence device is failing (action monitor) and it requires manual intervention to put it back to online state. imho there is a false assumption how we check ON state of device. It is tested purely on fence device, so if the device can turn ON/OFF empty plug we will not notice any problem. The fundamental issue can be boiled down to a simpler example, I think. Ignore IPMI fencing, lets assume the node has two PDUs for fencing (one for each PSU). At some point, a PDU is lost (destroyed, removed, whatever). At this point, the node is operational still but the ability to fence it has been lost. This requires human intervention to resolve, no debate there at all. We have a scenario though where the system may be entirely off the internet and is physically inaccessible, so realisation that the problem exists and/or the ability to repair it delayed. All of this is not a concern to developers, of course, but it sets the stage for how the issue can occur in the real world. So, later, the degraded node fails and a fence is issued. Of course, the fence will fail and the cluster will block, as it should. This is also not a concern. The issue is that, when the PDU fence occurs, it asks both PDUs to turn off the given outlets. The good node does this, but the other PDU is dead. This is where the issue arises. The good PDU is left in the 'off' position. So if the admin tries to manually (re)boot the node, s/he can't because there is no power from the good PDU. My proposal/request is that, if one device in a given method fails and the fence action was 'reboot', then re-enabling the port before moving on to the next fence device (if any) would help. Specifically because a 'fence_ack_manual' (or the pcmk equiv.) would not call an on. This is fundamentally not a cluster software problem, but more of a feature request, I suppose. It would save the admin some head-scratching. :) Great explanation, I got it. * 2x PSU/PDU (A,B) for one node * remove A -> connection timeout -> every action will fail * attempt to fence node -> planned action A-off,A-verify,B-off,B-verify,A-on,A-verify,B-on,B-verify * A-off will fail; B-off will turn PSU off and in such case you want to turn B-on. From fencing perspective, removing one PSU is not enough to think that machine is fenced so it does not matter if B is on or off. If this would be a configurable option, it should be fine. When you have one PDU with two different plugs for that node, this situation can be handled completely by fence agent. In such case - the first fail will end fence agent. So such fallback action should be done by someone from higher level (fenced / stonithd) probably. > From fencing perspective, removing one PSU is not enough to think that machine is fenced so it does not matter if B is on or off.
From a fencing perspective, I 100% agree. It's a failed fence attempt. The request is only to send an 'on' command to the device(s) that worked before moving on to the next fence method.
This is simply to leave the target node in a more recoverable state after, say, the user confirm's the node's death via fence_ack_manual. As it stands now, the admin will have to first realize the PDU outlet is off and second, turn it back on (which the admin might not be familiar with how to do that).
@Andrew: What do you think? @Andrew: What do you think? Agreed that it would have to happen at a higher level, the agent has nowhere near enough context. A Pacemaker based system would at least have noticed and notified the admin that one of the PDUs was dead - so hopefully there is time to rectify that problem before a node dies anyway. There is also the question as to whether a node that was fenced should be allowed to come back with only one PDU. On balance, I would not be in favor of adding another error path to cover this scenario. Fencing is primarily optimized around taking nodes down (or cutting them off) for safety/consistency, having them come back up again has to take a lower priority. I worry more about scenarios that could lead to a node being on when it should be off than the reverse :-) In an appliance model, could you not add "turn all the PDUs on" to your "start node X" command? I'm inclined to agree with Andrew on this. In HA, nothing should trump simplicity, particularly in the core. For this reason, I think we should close this as either WONTFIX or NOTABUG. If I want to, I can have our system handle this. ok, closing then. |