Bug 229650

Ok, this will require additional configuration information, because currently,
resources have an "all children alive" requirement.  That is, if any parent
resource has a child which is not in proper operational status, the parent is
considered broken as well.

This, of course, makes status checks easy: the service is broken if anything in
the service is broken

What we need is a way to have rgmanager ignore errors if they're immediately
correctable on a per-resource basis.  This is like the "recovery" flag -
however, the "recovery" operation is not allowed to affect *any* other resources
 - even if an explicit parent/child or other dependency exists.  So, "recovery"
will not solve the problem work if a resource has children.

So, what we basically need is a special attribute which can be assigned to
any/all resources which says (basically):

"This subtree is not dependent on its siblings, and can be safely restarted
without the parent being considered broken".

So, to expand on your example:

<service>
	<oracle />
	<ip special_new_attr="1">
		<script/>
	</ip>
</service>

If the IP address winds up missing, the script is stopped and the IP is stopped,
then restarted.  If oracle winds up broken, *everything* is restarted.  To make
them completely independent:

<service>
	<oracle special_new_attr="1"/>
	<ip special_new_attr="1">
		<script/>
	</ip>
</service>

This would work at all levels, too:

<service>
        <fs special_new_attr="1">
                <nfsexport>
                        <nfsclient/>
                </nfsexport>
        </fs>
        <fs special_new_attr="1">
                <nfsexport>
                        <nfsclient/>
                </nfsexport>
        </fs>
        <ip/>
</service>

The above example is an nfs service.  The two file system trees can be restarted
independently, but if the IP fails, everything must be restarted (Note: This
might not work for NFS services due to lock reclaim broadcasts, so is just there
for illustrative purposes).

As for maxinstances, instances of shared resources are expected to be able to
operate independently.  That is, if one instance fails, it does not imply that
they all have failed.  If it does, something is broken in the resource agent
and/or rgmanager.  If it isn't possible to make the resource instances
completely independent of one-another, then the resource agent should not define
maxinstances > 1.

<service>
        <fs name="one" special_new_attr="1">
                <nfsexport>
                        <nfsclient/>
                </nfsexport>
        </fs>
        <fs name="two" special_new_attr="1">
                <nfsexport>
                        <nfsclient/>
                </nfsexport>
                <script name="scooby"/>
        </fs>
        <ip/>
</service>

In this example, we add a script fs resource named "two".  If "two" fails, all
of its children must be restarted.  That is, the nfsxeport (and client) and the
script named "scooby" are all restarted.  Similarly, adhering to current
rgmanager behavior, if any of "two"'s children fail, everything up to and
including "two" will be restarted.  For example, if "scooby" fails, the
nfsexport/nfsclient children of "two" will also be restarted - and so will "two"
itself.

However, the file system "one" will never be affected by any of "two"'s problems.

Created attachment 148657 [details]
Patch that will not handle the special flag discussed in comments #1 and #2.

Hi,

as requested on IRC I attacched an initial patch against CVS HEAD done before
considerations in comments #1 and #2. This patch will ALWAYS stop resources
until the failed one, and, then starting from the failed one.

When the status on a resource fails the flags RF_NEEDSTOP and RF_NEEDSTART of
that node are setted.
Two new functions svc_condstart and svc_condstop are added.

The call to svc_stop was moved inside handle_recover_req, sp it will first check
on the recovery policies and, if needed, calls svc_condstop insted of svc_stop.

Thanks!

Thanks Simone - I won't be able to look at this in detail for about a week or
so.  Sorry (in advance) for the delay!

Fixing Product Name.  Cluster Suite was integrated into the Enterprise Linux for
version 5.0.

Created attachment 155385 [details]
rhel5 patch

This is a simplified patch which allows independent subtrees to be restarted as
part of a status check.

Created attachment 155386 [details]
Example script which causes a failed status check

Created attachment 155387 [details]
Example script which causes a successful status check

Created attachment 155388 [details]
Example config 1

Standard behavior.  Status fails w/o __independent_subtree flag, so the whole
service is noted as failed.

Created attachment 155389 [details]
Example config 2

Example config #2:  the script tag referencing the 'status-fail' script has
__independent_subtree="1".  This causes the script to be restarted inline as
part of the status check.  If the restart of the independent tree(s) fail, the
service is failed, and normal recovery takes place.

This makes __independent_subtree a bit like the "recover" action for resources,
except that children of a node marked __indepdent_subtree are also restarted
(the parent->child dependency relationship is **NOT** affected by the
__independent_subtree flag)

[lhh@ayanami daemons]$ ./rg_test test ./cluster.conf-example1 status service test00
malloc_init: Warning: using unlocked memory pages (got root?)
Running in test mode.
Checking status of test00...
<info>   Executing /tmp/status-fail status
[script] /tmp/status-fail status
<err>    script:foo: status of /tmp/status-fail failed (returned 1)
Status check of test00 failed

(if this were rgmanager, service recovery would commence)


[lhh@ayanami daemons]$ ./rg_test test ./cluster.conf-example2 status service test00
malloc_init: Warning: using unlocked memory pages (got root?)
Running in test mode.
Checking status of test00...
<info>   Executing /tmp/status-fail status
[script] /tmp/status-fail status
<err>    script:foo: status of /tmp/status-fail failed (returned 1)
<info>   Executing /tmp/status-success status
Node script:foo - CONDSTOP
<info>   Executing /tmp/status-fail stop
[script] /tmp/status-fail stop
Node script:foo - CONDSTART
<info>   Executing /tmp/status-fail start
[script] /tmp/status-fail start
Status of test00 is good

... note inline recovery.

Example log messages:

May 24 15:55:10 ayanami rg_test: [13666]: <err> script:foo: status of
/tmp/status-fail failed (returned 1) 
May 24 15:55:10 ayanami rg_test[13666]: <notice> status on script "foo" returned
1 (generic error) 
May 24 15:55:10 ayanami rg_test[13666]: <warning> Some independent resources in
service:test00 failed; Attempting inline recovery 
May 24 15:55:10 ayanami rg_test[13666]: <notice> Inline recovery of
service:test00 successful 

Created attachment 155835 [details]
Updated patch; fixes corner case

This updated patch fixes a corner case where if a child of an independent
subtree node failed, the whole service was restarted.  Now just the independent
subtree is restarted.

Created attachment 156923 [details]
ancillary patch

Patches in CVS

this doesn't work with non-independent subtrees sometimes.

I am testing a patch.

Created attachment 181341 [details]
Fix restart bug

I unit tested the patch with:

  <script name="logger" file="/log-me.sh"/>

And a service like this:

  <service autostart="1" exclusive="0" name="badsvc" recovery="restart">
    <script file="/test-script.sh" name="false" __independent_subtree="0">
      <script ref="logger"/>
    </script>
    <script ref="logger"/>

Contents of /log-me.sh:

#!/bin/sh
. /usr/share/cluster/ocf-shellfuncs
ocf_log notice "$0 $1"
exit 0

Contents of /test-script.sh:

#!/bin/sh
[ -f "/tmp/test-script-$1" ] && exit 1
exit 0

How it works:

(1) When you create /tmp/test-script-status, the script will fail that phase.
(2) If you have __independent_subtree="1" in cluster.conf for the above service
you will only see log-me.sh stop/start once (the child of the test-script is
restarted).
(3) If you have __independent_subtree="0" in cluster.conf for the above service,
you will see log-me.sh stop/start twice, because the whole service is considered
failed.

Unit-tested against migratory services (a la VMs) in order to ensure that
non-cluster-induced migration was still detected correctly; it was.  After
migrating (manually, using 'xm migrate') guest1 to et-virt05 (node 1 in
cluster.conf):

Aug 30 12:27:42 et-virt06 clurgmgrd[5112]: <notice> status on vm "guest1"
returned 1 (generic error) 
Aug 30 12:27:42 et-virt06 clurgmgrd[5112]: <notice> Migration: vm:guest1 is
running on 1 

More-preferred migration still works as expected, though an erroneous status
error showed up in the logs:

Aug 30 12:28:27 et-virt06 clurgmgrd[5112]: <info> State change:
et-virt07.lab.boston.redhat.com UP 
Aug 30 12:28:28 et-virt06 clurgmgrd[5112]: <notice> Migrating vm:guest3 to
better node et-virt07.lab.boston.redhat.com 
Aug 30 12:28:28 et-virt06 clurgmgrd[5112]: <notice> Migrating vm:guest3 to
et-virt07.lab.boston.redhat.com 
Aug 30 12:28:34 et-virt06 clurgmgrd[5112]: <notice> Migration of vm:guest3 to
et-virt07.lab.boston.redhat.com completed 
vvvvvv
Aug 30 12:28:35 et-virt06 clurgmgrd[5112]: <notice> status on vm "guest3"
returned 1 (generic error) 
^^^^^^

This is considered a separate, non-critical bug.

I noticed, using the RHEL51 branch that with a config like this:

<service autostart="1" domain="argle" name="test00">
    <script name="foo01" __independent_subtree="1" file="/tmp/script01">
        <script name="foo02" __independent_subtree="0" file="/tmp/script02"/>
    <script name="foo03" __independent_subtree="0" file="/tmp/script03"/>
    </script>
</service>

if foo02 fails then the whole service is restarted instead of only restarting
from foo01.

I did a little patch that will try to fix this problem. I hope the logic I
followed is correct.

Thanks!

Created attachment 205561 [details]
Restart only independent subtrees if a non independent child fails.

Simone, your patch is correct.

I think the return SFL_RECOVERABLE should be rv = SFL_RECOVERABLE

Created attachment 205861 [details]
Updated patch

Slight change; set rv instead of returning.  Functionally, there's no change;
this is in case we add other stuff to the function later.

I did some tests with the latest patch (id=205861) using various combinations of
resources and differents __independent_subtree values and it looks ok.

Ex:

<service autostart="1" domain="argle" name="test00">
    <script name="foo01" __independent_subtree="1" file="/tmp/script01">
        <script name="foo02" __independent_subtree="0" file="/tmp/script02"/>
        <script name="foo03" __independent_subtree="1" file="/tmp/script03"/>
    </script>
    <script name="foo04" __independent_subtree="0" file="/tmp/script04"/> 
</service>

fail foo01 : foo03, foo02, foo01 stopped; foo01, foo02, foo03 started;
fail foo02 : foo03, foo02, foo01 stopped; foo01, foo02, foo03 started;
fail foo03 : foo03 stopped; foo03 started;
fail foo04 : whole service restarted

Thanks!

An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on the solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2007-0580.html