Bug 510722
| Summary: | Dropped packet simulation causes clustered broker nodes to exit due to inconsistent errors | ||
|---|---|---|---|
| Product: | Red Hat Enterprise MRG | Reporter: | Gordon Sim <gsim> |
| Component: | qpid-cpp | Assignee: | messaging-bugs <messaging-bugs> |
| Status: | CLOSED NOTABUG | QA Contact: | MRG Quality Engineering <mrgqe-bugs> |
| Severity: | high | Docs Contact: | |
| Priority: | urgent | ||
| Version: | 1.1.1 | CC: | cctrieloff, rattapat+nobody |
| Target Milestone: | 1.1.6 | ||
| Target Release: | --- | ||
| Hardware: | All | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2009-07-11 23:44:19 UTC | Type: | --- |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
The rpms version is -18 They simulated the packet drop case using IPTables and DROP. They dropped packets on both ways on a particular bond. At the time the test was run there were several systems. Some of these systems would have been idle due to not receiving messages (they send messages only when they receive). So some of these systems may have used heartbeats. ***They also dropped packets between nodes in the cluster***. This may have been the reason why some nodes exited. They did this in stages. They dropped packets between broker and engine. Then they dropped packets between the nodes in the cluster. All this while some applications were still sending and receiving messages from the brokers. Hope this info helps {***They also dropped packets between nodes in the cluster***. This may have
been the reason why some nodes exited.)
If the network on AIS is broken, then this is a slip brain simulation. In this case the brokers will not agree on state when rejoining if a quoram device is not used to pro-actively shut down brokers. On reconnect, some brokers will exist to create a set of brokers that are consistent. Shut down brokers may be restarted, and will be updated.
Note that redundant multicast rings, or bonding can be used to guard against cluster network failures.
Marking as NOTABUG
|
Description of problem: Package loss was simulated using iptables to drop packets during use of a 4 node cluster. This included: - a 2 seconds network outage causing 15% message loss - a 9 seconds network outage causing 30% message loss - a 12 second outage causing 85% message loss Three of the four qpidd processes then exited with errors similar to the following: 2009-jul-02 14:42:10 error Channel exception: not-attached: receiving Frame[BEbe; channel=4; {ExecutionSyncBody: }]: channel 4 is not attached (qpid/amqp_0_10/SessionHandler.cpp:79) 2009-jul-02 14:42:10 error Channel exception: not-attached: receiving Frame[Bbe; channel=3; {MessageTransferBody: destination=XXX; accept-mode=1; acquire-mode=0; }]: channel 3 is not attached (qpid/amqp_0_10/SessionHandler.cpp:79) 2009-jul-02 14:42:10 error Channel exception: not-attached: receiving Frame[be; channel=3; header (36 bytes); properties={{MessageProperties: content-length=158; }{DeliveryProperties: routing-key=YYY; }}]: channel 3 is not attached (qpid/amqp_0_10/SessionHandler.cpp:79) 2009-jul-02 14:42:10 critical 10.142.186.5:28225(READY/error) Error 29857411 did not occur on 10.142.186.3:3843 2009-jul-02 14:42:10 error Error delivering frames: Aborted by local failure that did not occur on all replicas 2009-jul-02 14:42:10 notice 10.142.186.5:28225(LEFT/error) leaving cluster qgwbm_5672 2009-jul-02 14:42:10 notice Shut down