Bug 1068926

Created attachment 875129 [details]
screenshot

1- Have 2 hosts in iscsi data centre in up state
2. put hsm host in maintenance and block traffic to storage domain
3. put host back to up

result: 
Host goes to state unassigned for ~2minutes and then goes to non operational state.

However, in the event log tab in the management you can see an event saying: 
State was set to Up for host monique-vds01.tlv.redhat.com 

after an the event: Host monique-vds01.tlv.redhat.com was activated by admin@internal

Created attachment 875130 [details]
engine.log

failqa per comment 2

Hi Tareq,

I can see in the log that:
1. Host was switched to unassigned
2. Host was switched to up
3. Host was switched to non operational

Is that what you see in the audit log too? I think this is the expected behaviour, no?

No on the audit log i can see the "State was set to up for Host monique-vds01.tlv.redhat.com"
This message it wrong. What really happens is that host goes into unassigned then goes into non operational. 

You need to omit this message from event log .. because it is wrong and confusing.

Got a strange behaviour, not sure if it's the expected one:

1. added two hosts to iscsi DC-CLUSTER_SD
2. both hosts are connected to storage and up
3. put hsm to maintenance state
4. on hsm: iptables -A OUTPUT -p tcp --dport 3260 -j DROP
5. at this point hsm became 'unassigned' for 2 minutes, then non-operational, and after 3 more minutes hsm went back to 'up' state for few seconds then started another "loop" of the same unassigned-non-operational-up-etc....
only after I set iptables to allow connection to storage again I was able to put hsm to 'up' state.

I'll attach my engine.log and a snapshot of the auditlog.

Created attachment 881863 [details]
engine log - 02.04.14

Created attachment 881864 [details]
screenshot - 02.04.14

This bug is currently attached to errata RHBA-2014:17286. If this change is not to be documented in the text for this errata please either remove it from the errata, set the requires_doc_text flag to minus (-), or leave a "Doc Text" value of "--no tech note required" if you do not have permission to alter the flag.

Otherwise to aid in the development of relevant and accurate release documentation, please fill out the "Doc Text" field above with these four (4) pieces of information:

* Cause: What actions or circumstances cause this bug to present.

* Consequence: What happens when the bug presents.

* Fix: What was done to fix the bug.

* Result: What now happens when the actions or circumstances above occur. (NB: this is not the same as 'the bug doesn't present anymore')

Once filled out, please set the "Doc Type" field to the appropriate value for the type of change made and submit your edits to the bug.

For further details on the Cause, Consequence, Fix, Result format please refer to:

https://bugzilla.redhat.com/page.cgi?id=fields.html#cf_release_notes

Thanks in advance.

(In reply to sefi litmanovich from comment #8)
> Got a strange behaviour, not sure if it's the expected one:
> 
> 1. added two hosts to iscsi DC-CLUSTER_SD
> 2. both hosts are connected to storage and up
> 3. put hsm to maintenance state
> 4. on hsm: iptables -A OUTPUT -p tcp --dport 3260 -j DROP
> 5. at this point hsm became 'unassigned' for 2 minutes, then
> non-operational, and after 3 more minutes hsm went back to 'up' state for
> few seconds then started another "loop" of the same
> unassigned-non-operational-up-etc....
> only after I set iptables to allow connection to storage again I was able to
> put hsm to 'up' state.
> 
> I'll attach my engine.log and a snapshot of the auditlog.

Please retry this test with REJECT rather than DROP

Retried just now with REJECT instead of DROP.

result is pretty much the same.
host becomes unassigned for about 2 minutes then engine tries to set host to up state (event log says:"State was set to up on host") it remains unassigned until it fails to connect the host to the storage, gives the expected message and moves host to non-operational. after few minutes engine tries to set host to up again and it becomes unassigned.
this loop can last forever it seems, this cannot be the expected result.

Sefi, can you send me an environment where you reproduced this bug? I am kind of shooting in the dark here...

(In reply to sefi litmanovich from comment #13)
> Retried just now with REJECT instead of DROP.
> 
> result is pretty much the same.
> host becomes unassigned for about 2 minutes then engine tries to set host to
> up state (event log says:"State was set to up on host") it remains
> unassigned until it fails to connect the host to the storage, gives the
> expected message and moves host to non-operational. after few minutes engine
> tries to set host to up again and it becomes unassigned.
> this loop can last forever it seems, this cannot be the expected result.

This behavior is the exact expected behaviour in these situations.

The host recovery mechanism wakes up every couple of minutes and tries to reconnect and bring everything to up.

So moving this bug back to ON_QA so you guys can move it to VERIFIED.

Verified according to according to comments 8,13,16.
using rhevm-3.3.2-0.50.el6ev.noarch

Was released with 3.3.2