This is easily reproduced.


Dec 22 11:55:05 taft-03 clurgmgrd: [7233]: <notice> Removing ownership tag (taft-03) from TAFT/ha1 
Dec 22 11:55:07 taft-03 clurgmgrd: [7233]: <err> Unable to delete tag from TAFT/ha1 
Dec 22 11:55:07 taft-03 clurgmgrd: [7233]: <err> Failed to stop TAFT/ha1 
Dec 22 11:55:07 taft-03 clurgmgrd: [7233]: <err> Failed to stop TAFT/ha1 
Dec 22 11:55:07 taft-03 clurgmgrd[7233]: <notice> stop on lvm "lvm" returned 1 (generic error) 
Dec 22 11:55:07 taft-03 clurgmgrd[7233]: <crit> #12: RG service:halvm failed to stop; intervention required 
Dec 22 11:55:07 taft-03 clurgmgrd[7233]: <notice> Service service:halvm is failed 
Dec 22 11:55:07 taft-03 clurgmgrd[7233]: <alert> #2: Service service:halvm returned failure code.  Last Owner: taft-03 
Dec 22 11:55:07 taft-03 clurgmgrd[7233]: <alert> #4: Administrator intervention required.

when I try to run revolution_9, it craps out saying,
"only supported IO load for HA LVM is NONE".  You seem to have managed to override this.  How?

I /think/ I understand what is going on here.  Since I have been unable to run the test, I've broken down what I think the steps are that are causing the problem:
1) add tag to a mirror ('lvchange --addtag foo vg/lv')
2) disable a mirror leg (with no I/O and after sync has completed)
3) run 'lvconvert --repair --use-policies vg/lv'
4) run 'lvchange --deltag foo vg/lv'

Step #4 now fails because you can't change the metadata of a VG (i.e. remove the tag on the LV) while there are missing PVs.  This comes up now because the dmeventd script has switched from using 'vgreduce --removemissing' to 'lvconvert --repair'.  The former would remove the missing PV from the VG while the latter does not.

The output from the 'lvchange --deltag' operation is suppressed by rgmanager - instead we get the generic "Unable to delete tag from vg/lv" message.

The solution would be to run 'vgreduce --removemissing' if the 'lvchange --deltag' operation fails and try a second time to remove the tag.  The solution is also simple to verify - if I can run the test.

Created attachment 551698 [details]
Patch to fix problem

Testing has revealed that the problem was correctly identified by comment #3.

The script was changed so that it could handle a particular failure case - something that was necessary due to the change in LVM behaviour.  Other operations (i.e. most failure cases and nominal operation) are unaffected.

Fix verified in rgmanager-2.0.52-28.el5.

Scenario kill_primary_leg: Kill primary leg

********* Mirror info for this scenario *********
* mirrors:            ha1
* leg devices:        /dev/sdc1 /dev/sdd1
* log devices:        /dev/sdh1
* failpv(s):          /dev/sdc1
* failnode(s):        taft-04
* leg fault policy:   allocate
* log fault policy:   allocate

* HA mirror:          YES
* HA service name:    halvm
* HA current owner:   taft-04
*************************************************

Mirror Structure(s):
  LV             Attr   LSize  Copy%  Devices                        
  ha1            mwi-ao 25.00G 100.00 ha1_mimage_0(0),ha1_mimage_1(0)
  [ha1_mimage_0] iwi-ao 25.00G        /dev/sdc1(0)                   
  [ha1_mimage_1] iwi-ao 25.00G        /dev/sdd1(0)                   
  [ha1_mlog]     lwi-ao  4.00M        /dev/sdh1(0)                   

PV=/dev/sdc1
        ha1_mimage_0: 5.1
PV=/dev/sdc1
        ha1_mimage_0: 5.1


Disabling device sdc on taft-04

Attempting I/O to cause mirror down conversion(s) on taft-04
1+0 records in
1+0 records out
512 bytes (512 B) copied, 3.8e-05 seconds, 13.5 MB/s

RELOCATING halvm FROM taft-04 TO taft-01

Verifying current sanity of lvm after the failure

Mirror Structure(s):
  LV             Attr   LSize  Copy%  Devices                        
  ha1            mwi-ao 25.00G   4.97 ha1_mimage_0(0),ha1_mimage_1(0)
  [ha1_mimage_0] Iwi-ao 25.00G        /dev/sdd1(0)                   
  [ha1_mimage_1] Iwi-ao 25.00G        /dev/sde1(0)                   
  [ha1_mlog]     lwi-ao  4.00M        /dev/sdh1(1)                   

Verify that each of the mirror repairs finished successfully

Verifying FAILED device /dev/sdc1 is *NOT* in the volume(s)
olog: 1
Verifying LOG device(s) /dev/sdh1 *ARE* in the mirror(s)
Verifying LEG device /dev/sdd1 *IS* in the volume(s)
verify the newly allocated dm devices were added as a result of the failures
Checking EXISTENCE of ha1_mimage_0 on:  taft-01
Checking EXISTENCE of ha1_mimage_1 on:  taft-01

Enabling device sdc on taft-04


Waiting until all mirrors become fully syncd...
   0/1 mirror(s) are fully synced: ( 6.66% )
   0/1 mirror(s) are fully synced: ( 8.07% )
   [...]
   0/1 mirror(s) are fully synced: ( 98.72% )
   1/1 mirror(s) are fully synced: ( 100.00% )



Jan 11 11:30:37 taft-01 clurgmgrd: [11593]: <notice> Removing ownership tag (taft-01) from TAFT/ha1 
Jan 11 11:30:39 taft-01 clurgmgrd: [11593]: <err> Unable to delete tag from TAFT/ha1 
Jan 11 11:30:39 taft-01 clurgmgrd: [11593]: <err> Attempting volume group clean-up and retry

    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
Cause:

Consequence:

Fix:

Result:

    Technical note updated. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    Diffed Contents:
@@ -1,7 +1,11 @@
 Cause:
+The behaviour of the LVM commands has changed, causing failed devices to be removed from a mirror but not from the volume group as it was previously.
 
 Consequence:
+Volume groups cannot be changed while a failed device is present.  This includes changing the tags on a logical volume.  Failure to alter the tags on the affected logical volume causes an inability to relocate the service.
 
 Fix:
+An additional LVM command is called to remove the failed physical volume from the volume group.  The tags on the logical volumes can then be altered.
 
-Result:+Result:
+Two commands are now required to perform the operations that one command did in the past.  This change resolves the issue.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2012-0163.html