Bug 1601561
| Summary: | [OSP13][z-stream]Cannot launch instance after controller replacement - "VirtualInterfaceCreateException: Virtual Interface creation failed" | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Artem Hrechanychenko <ahrechan> |
| Component: | rabbitmq-server | Assignee: | John Eckersberg <jeckersb> |
| Status: | CLOSED CURRENTRELEASE | QA Contact: | pkomarov |
| Severity: | high | Docs Contact: | |
| Priority: | high | ||
| Version: | 13.0 (Queens) | CC: | apevec, chjones, dasmith, dcadzow, eglynn, jeckersb, jhakimra, kchamart, knoha, lhh, lmiccini, lyarwood, michele, ohochman, plemenko, sbauza, sgordon, srevivo, vromanso |
| Target Milestone: | z2 | Keywords: | Triaged, ZStream |
| Target Release: | 13.0 (Queens) | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2020-03-10 15:30:27 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
Artem Hrechanychenko
2018-07-16 16:54:57 UTC
The reports should be available here: http://rhos-release.virt.bos.redhat.com/log/bz1601561 Somehow controller-2 has gotten into a cluster by itself:
()[root@controller-2 /]# rabbitmqctl cluster_status
Cluster status of node 'rabbit@controller-2'
[{nodes,[{disc,['rabbit@controller-2']}]},
{running_nodes,['rabbit@controller-2']},
{cluster_name,<<"rabbit">>},
{partitions,[]},
{alarms,[{'rabbit@controller-2',[]}]}]
()[root@controller-2 /]# echo $?
0
And the resource agent just checks cluster_status rc for the monitor action, so since it returns 0 it thinks everything is ok.
Meanwhile, 0 and 3 are clustered fine:
()[root@controller-0 /]# rabbitmqctl cluster_status
Cluster status of node 'rabbit@controller-0'
[{nodes,[{disc,['rabbit@controller-0','rabbit@controller-3']}]},
{running_nodes,['rabbit@controller-3','rabbit@controller-0']},
{cluster_name,<<"rabbit">>},
{partitions,[]},
{alarms,[{'rabbit@controller-3',[]},{'rabbit@controller-0',[]}]}]
Unsurprising that instance launch fails in this state.
I just did `pkill -9 beam.smp` on controller-2 to force pacemaker to fail it and restart, and it joined the cluster correctly after that:
()[root@controller-2 /]# rabbitmqctl cluster_status
Cluster status of node 'rabbit@controller-2'
[{nodes,[{disc,['rabbit@controller-0','rabbit@controller-2',
'rabbit@controller-3']}]},
{running_nodes,['rabbit@controller-0','rabbit@controller-3',
'rabbit@controller-2']},
{cluster_name,<<"rabbit">>},
{partitions,[]},
{alarms,[{'rabbit@controller-0',[]},
{'rabbit@controller-3',[]},
{'rabbit@controller-2',[]}]}]
So two things here:
1. It never should have gotten into this state in the first place, so we should try to figure out how exactly it happened. Might not be so easy due to timing issues.
2. We have enough information in the resource agent to know how many nodes are started, so we can modify the health check to make sure each node is clustered, *and* it's clustered with the correct number of running nodes. Probably should be safe and make sure that action fails like 3x with 5s sleep so we don't catch false mismatches if the cluster is in the process of transitioning. It should settle within 15s.
Hello John, Is 'pkill -9 beam.smp' on a failure node a workaround for this issue at this moment? Best Regards, Keigo Noha (In reply to Keigo Noha from comment #6) > Hello John, > > Is 'pkill -9 beam.smp' on a failure node a workaround for this issue at this > moment? > > Best Regards, > Keigo Noha For a workaround, better is to just restart rabbitmq-bundle with pcs. I only killed the bad one to demonstrate that a failed monitor would allow pacemaker to restart the service and it would correctly rejoin the cluster. (In reply to John Eckersberg from comment #8) > (In reply to Keigo Noha from comment #6) Artem can you please validate the suggested W/A and see if it will allow to finish the controller replacement procedure and get the system back to a fully stable & working state. Didn't reproduced from first attempt. will try another one Hello All! Could someone please try to reproduce it with the latest resource-agents? Artem? we assume this has been fixed by the latest resource agents. please reopen if required. |