Hide Forgot
Description of problem: The following mirrored volume was exclusively active on taft-02 only. # Test clean up output: Stopping the io load (collie/xdoio) on mirror(s) Unmounting gfs and removing mnt point on taft-02... Deactivating and removing mirror(s) Checking for left over mirror dm devices on ...taft-02 all mirror images for syncd_secondary_4legs_1 weren't removed from d-m on taft-02 [root@taft-02 ~]# lvs LV VG Attr LSize Origin Snap% Move Log Copy% Convert LogVol00 VolGroup00 -wi-ao 58.38G LogVol01 VolGroup00 -wi-ao 9.75G [root@taft-02 ~]# dmsetup ls helter_skelter-syncd_secondary_4legs_1_mimage_4 (253, 8) helter_skelter-syncd_secondary_4legs_1_mimage_3 (253, 6) helter_skelter-syncd_secondary_4legs_1_mimage_2 (253, 5) helter_skelter-syncd_secondary_4legs_1_mimage_0 (253, 3) helter_skelter-syncd_secondary_4legs_1_mlog (253, 2) VolGroup00-LogVol01 (253, 1) VolGroup00-LogVol00 (253, 0) Version-Release number of selected component (if applicable): 2.6.18-256.el5 lvm2-2.02.84-3.el5 BUILT: Wed Apr 27 03:42:24 CDT 2011 lvm2-cluster-2.02.84-3.el5 BUILT: Wed Apr 27 03:42:43 CDT 2011 device-mapper-1.02.63-2.el5 BUILT: Fri Mar 4 10:23:17 CST 2011 device-mapper-event-1.02.63-2.el5 BUILT: Fri Mar 4 10:23:17 CST 2011 cmirror-1.1.39-10.el5 BUILT: Wed Sep 8 16:32:05 CDT 2010 kmod-cmirror-0.1.22-3.el5 BUILT: Tue Dec 22 13:39:47 CST 2009
Could you please attach traces how have you get into this situation? How did you setup your exclusively activated device?
The key here is to exclusively activate the volume on one node (NODE A) and then deactivate and remove on another node (NODE B). In rhel6.1, the deactivation and removal on NODE B, will cause the dm devices to be removed on NODE A. However in rhel5.7, that's not the case. The devices are left over on NODE A. I'd argue that you should not even be allowed to deactivate and remove a device that's exclusively activated on another node.
The cluster can be managed from a single node: deactivate/remove from any node is an important part of that. Two things we might do: 1. Give a "LV is active on other node(s) - are you sure?" prompt? 2. Provide configurable options that allow manipulation to be restricted to certain nodes. I don't find the first very practical when you think through the various combinations and conflicting ways it ought to behave in different configurations. I think the second has been discussed before and is the better solution. As to this particular bug, if I read your comment correctly, you're indicating this is fixed in RHEL6.1 but not in 5.7 so hopefully it is something we can backport.
Correct, in RHEL6.1 there are no left over dm devices when the mirror is removed from a node other than the one which has it activated exclusively.
It is cmirror or just only local mirror? Note that cmirror package is separate in RHEL5 (and I think it was not updated.)
Corey, what exaclty need to bve done to reproduce it? (in the terms of lvcreate/lvchange/lvremove commands) I created clustered mirror on node A, and deactivated + removed from node B - and it removed all active dm devices from node A. So it seems to work for me...
I can no longer reproduce this with just the commands listed in comment #2. I can however still reproduce this by running the test helter_skelter. So, it looks like this still involves exclusively activated mirrors, and may also involve device failure, I/O, or both. I'll attempt to better boil this down...
ok, I am setting needinfo for now, we need reproducer...
*** Bug 637298 has been marked as a duplicate of this bug. ***
I'm unable to reproduce this w/ the latest 5.9 tree. Closing... ./helter_skelter -l /usr/tests/sts-rhel5.8 -r /usr/tests/sts-rhel5.8 -R ../../var/share/resource_files/taft.xml -E taft-04 2.6.18-333.el5 lvm2-2.02.88-9.el5 BUILT: Wed Jul 25 10:13:00 CDT 2012 lvm2-cluster-2.02.88-9.el5 BUILT: Wed Jul 25 10:09:34 CDT 2012 device-mapper-1.02.67-2.el5 BUILT: Mon Oct 17 08:31:56 CDT 2011 device-mapper-event-1.02.67-2.el5 BUILT: Mon Oct 17 08:31:56 CDT 2011 cmirror-1.1.39-15.el5 BUILT: Thu Apr 26 17:07:01 CDT 2012 kmod-cmirror-0.1.22-3.el5 BUILT: Tue Dec 22 13:39:47 CST 2009