Created attachment 1591341 [details]
Description of problem:
- Start 3 EAP 6.4 pods in Openshift in cluster
-- deploy singleton timer inserts new line to DB every 10 seconds (it will happen only on one pod)
- between 10 sec interval of timer get number rows in DB over servlet, expect that it increases by 1 every 10 seconds
Version-Release number of selected component (if applicable):
Sometimes happens that 2 server are working as singleton masters and both of them are inserting rows into DB
Number of lines increases by 1 every 10 seconds. There is just singleton master.
Attaching logs from test and pods (logs.zip) - name of tests HASingletonCounterTest.haSingletonCanCountForTwoMinutesTest
Looking at the logs there is visible that pod sinc-1-9q5vs elected itself as singleton provider. Then around 2sec later it elected pod hsinc-1-mvmmx as singleton provider.
At this moment pod hsinc-1-mvmmx was restarted (most likely because of failing liveness probe).
However 3rd pod hsinc-1-g4tmc just started and was elected as singleton provider as well.
So at this moment sinc-1-9q5vs and hsinc-1-g4tmc works as singleton master which is wrong.