Description of problem:  When relocating a service, rgmanager will try out the best nodes first and if all else fails, will restart the service locally (even if it wasn't started to begin with).  However, it doesn't actually check to see whether the local node is eligible to run the service, and so you can end up with a service in a restricted domain starting on a member outside that domain.

A customer encountered this in the only way I was able to conceive that this could happen, and admittedly the conditions under which its possible are fairly narrow.  If the service is in a stopped state, and you issue a 'clusvcadm -r <service>' from a non-domain-member without specifying a destination, and all of the members of that domain fail to start the service, then eventually once all members are exhausted the node where the command was run will restart it locally.  If that happens to succeed there where it failed on the other nodes, it will stay running on this node which is not a member of the domain.


Version-Release number of selected component (if applicable): rgmanager-3.0.12.1-19.el6


How reproducible: Easily with contrived conditions, probably rarely in the wild


Steps to Reproduce:
1. Create a restricted failoverdomain with one node left out.

2. Create a service that can be configured to fail on-demand on startup.  I use a script resource that fails on start with the presence of a special file, and succeeds otherwise for everything else. Put in the failover domain.

3. Configure all nodes in the failoverdomain to fail the service on start.

4. Put the service into a stopped state:

  # clusvcadm -s <service>

5. On the non-domain-member, run 'clusvcadm -r <service>'

Actual results: 

		<failoverdomains>
			<failoverdomain name="1then2" nofailback="1" ordered="1" restricted="1">
				<failoverdomainnode name="rhel6-node1.example.com" priority="1"/>
				<failoverdomainnode name="rhel6-node2.example.com" priority="2"/>
			</failoverdomain>
			<failoverdomain name="2then1" nofailback="1" ordered="1" restricted="1">
				<failoverdomainnode name="rhel6-node1.example.com" priority="2"/>
				<failoverdomainnode name="rhel6-node2.example.com" priority="1"/>
			</failoverdomain>
		</failoverdomains>
		<service domain="1then2" name="test">
			<script file="/root/script" name="script"/>
		</service>

# clusvcadm -r test
Trying to relocate service:test...Failed; service running on original owner

# clustat
Cluster Status for rhel6-cluster @ Thu Oct  9 14:19:49 2014
Member Status: Quorate

 Member Name                                                     ID   Status
 ------ ----                                                     ---- ------
 rhel6-node1.example.com                                             1 Online, rgmanager
 rhel6-node2.example.com                                             2 Online, rgmanager
 rhel6-node3.example.com                                             3 Online, Local, rgmanager

 Service Name                                                     Owner (Last)                                                     State         
 ------- ----                                                     ----- ------                                                     -----         
 service:test                                                     rhel6-node3.example.com                                          started  

Oct  9 14:18:37 rhel6-node1 rgmanager[27937]: Starting stopped service service:test
Oct  9 14:18:37 rhel6-node1 rgmanager[29176]: [script] Executing /root/script start
Oct  9 14:18:37 rhel6-node1 rgmanager[29197]: [script] script:script: start of /root/script failed (returned 1)
Oct  9 14:18:37 rhel6-node1 rgmanager[27937]: start on script "script" returned 1 (generic error)
Oct  9 14:18:37 rhel6-node1 rgmanager[27937]: #68: Failed to start service:test; return value: 1
Oct  9 14:18:37 rhel6-node1 rgmanager[27937]: Stopping service service:test
Oct  9 14:18:37 rhel6-node1 rgmanager[29226]: [script] Executing /root/script stop
Oct  9 14:18:37 rhel6-node1 rgmanager[27937]: Service service:test is recovering

Oct  9 14:18:10 rhel6-node2 rgmanager[13591]: Service service:test is recovering
Oct  9 14:18:37 rhel6-node2 rgmanager[13591]: Recovering failed service service:test
Oct  9 14:18:38 rhel6-node2 rgmanager[14616]: [script] Executing /root/script start
Oct  9 14:18:38 rhel6-node2 rgmanager[14637]: [script] script:script: start of /root/script failed (returned 1)
Oct  9 14:18:38 rhel6-node2 rgmanager[13591]: start on script "script" returned 1 (generic error)
Oct  9 14:18:38 rhel6-node2 rgmanager[13591]: #68: Failed to start service:test; return value: 1
Oct  9 14:18:38 rhel6-node2 rgmanager[13591]: Stopping service service:test
Oct  9 14:18:38 rhel6-node2 rgmanager[14666]: [script] Executing /root/script stop
Oct  9 14:18:38 rhel6-node2 rgmanager[13591]: Service service:test is recovering

Oct  9 14:18:40 rhel6-node3 rgmanager[8721]: #70: Failed to relocate service:test; restarting locally
Oct  9 14:18:40 rhel6-node3 rgmanager[8721]: Recovering failed service service:test
Oct  9 14:18:40 rhel6-node3 rgmanager[9559]: [script] Executing /root/script start
Oct  9 14:18:40 rhel6-node3 rgmanager[8721]: Service service:test started


Expected results: Service goes back to a stopped state if it can't start on any domain members, and the local node doesn't try to start it if its not in the domain.


Additional info: