Bug 147531

This isn't lock manager specific, the same thing happens on cman/dlm.

This is real easy to reproduce. 

1. Get a gfs mounted on all nodes in the cluster (I had 3 gfs). 
2. Reboot everyone
3. verify that the dm entries are still there:
   [root@tank-02 ~]# dmsetup ls
   gfs-gfs1        (253, 3)
   gfs-gfs0        (253, 2)
   VolGroup00-LogVol01     (253, 0)
   VolGroup00-LogVol00     (253, 1)
   gfs-gfs2        (253, 4)

4. Unactivate the vg, then remove the lv, vg, and pv. 
5. Verify that all lvms are deleted on each node.
6. The dm entries will still be there:
   [root@tank-02 ~]# dmsetup ls
   gfs-gfs1        (253, 3)
   gfs-gfs0        (253, 2)
   VolGroup00-LogVol01     (253, 0)
   VolGroup00-LogVol00     (253, 1)
   gfs-gfs2        (253, 4)

7. Now make a new pv, vg, and lv
8. make a filesystem out of the lv and mount.
9. The dm entries will still be there in addition to the newly created
one:
   [root@tank-02 ~]# dmsetup ls
   gfs-gfs1        (253, 3)
   gfs-gfs0        (253, 2)
   myvg-lvol0      (253, 5)
   VolGroup00-LogVol01     (253, 0)
   VolGroup00-LogVol00     (253, 1)
   gfs-gfs2        (253, 4)


With those entries there, nothing but LVM is able to do anything with
that storage.

10. If you reboot all the nodes again the first dm entries will go away:

[root@tank-05 ~]# dmsetup ls
myvg-lvol0      (253, 2)
VolGroup00-LogVol01     (253, 0)
VolGroup00-LogVol00     (253, 1)

Sorry, but I still don't understand what you're doing here.
 
Which of all those steps is the *first* one where something happens
that shouldn't?  Then home in on that step to see what's going wrong
and provide more details of what you're running and what output you
get.  Everything after the first place things go wrong is not interesting.

For example, is step 2 -> step 3 what you expected? I can't tell from
the information so far if it is.  Then step 5 to step 6 certainly
doesn't make any sense to me - if the 'lvms' are deleted on each node,
dmsetup surely won't show anything?  So is it step 4 or step 5 where
something is wrong?

"Then step 5 to step 6 certainly doesn't make any sense to me - if the
'lvms' are deleted on each node, dmsetup surely won't show anything?"

Exactly, but it does. :) hence this bug. That is where I expected the
dm entries to be gone (after step 5), but they remain. 

So please document what you're doing in those steps: correct status
before, the one command you run that does the wrong thing, status
after it.

The command which is supposed to do the deletion of the dm entires is
the deactivation of the VG. So before doing the 'vgchange -an' you
will still see the dm entries, and afterwards you should not. 

Out of the blue this morning, I was nolonger able to reproduce this
issue. The same issue which us testers have been seeing at will for
the past sennight. 

Now, this afternoon, when trying to reproduce this, the lvdisplay cmds
hang.

.
.
.
stat64("/dev/sda2", {st_mode=S_IFBLK|0660, st_rdev=makedev(8, 2),
...}) = 0
stat64("/dev/sda2", {st_mode=S_IFBLK|0660, st_rdev=makedev(8, 2),
...}) = 0
open("/dev/sda2", O_RDONLY|O_DIRECT|O_LARGEFILE|0x40000) = 4
fstat64(4, {st_mode=S_IFBLK|0660, st_rdev=makedev(8, 2), ...}) = 0
ioctl(4, BLKBSZGET, 0x999e4f8)          = 0
_llseek(4, 2048, [2048], SEEK_SET)      = 0
read(4, "\312\213Q. LVM2 x[5A%r0N*>\1\0\0\0\0\10\0\0\0\0\0\0"...,
1024) = 1024
_llseek(4, 20480, [20480], SEEK_SET)    = 0
read(4, "gfs {\nid = \"lYOiwj-NkXU-hnJX-fLR"..., 1024) = 1024
close(4)                                = 0
stat64("/proc/lvm/VGs/gfs", 0xbff148e0) = -1 ENOENT (No such file or
directory)
rt_sigprocmask(SIG_SETMASK, ~[RTMIN RT_1], [], 8) = 0
write(3, "3\1\30\0\0\0\0\0\0\0\0\0\10\0\0\0\0\1\0V_gfs\0\0", 26) = 26
read(3,


<occupational hypnotherapist> That's messed up. </<occupational
hypnotherapist>

It turns out that the lvdisplay cmd was hanging as a result of all the
dm entries I had.  

I originally had 9 gfs filesystem mounted on all nodes, and I then
rebooted all of them to try and reproduce this issuse.  When the nodes
came back up I saw the 9 dm entries and started ccsd, gulmd, clvmd,
and then attempted an lvdisplay which would hang. So I rebooted again
and tried the exact same above senario. I repeated this 4 or 5 times,
each time resulting in hung lv cmds, until I decided to remove the dm
entries with 'dmsetup remove' and at which point everything was fine
again. 

This bug appears to have morphed in the past 24 hours.
Now any node with dm entries after a reboot ends up with hung lvm cmds.

Node WITHOUT dm entries:

[root@tank-01 ~]# cat /proc/partitions
major minor  #blocks  name
   3     0   39082680 hda
   3     1     104391 hda1
   3     2   38973690 hda2
 253     0    1048576 dm-0
 253     1   37912576 dm-1
   8     0 1367490560 sda
   8     1  683742464 sda1
   8     2  683742465 sda2
[root@tank-01 ~]# dmsetup ls
VolGroup00-LogVol01     (253, 0)
VolGroup00-LogVol00     (253, 1)
[root@tank-01 ~]# lvdisplay
  --- Logical volume ---
  LV Name                /dev/gfs/gfs0
  VG Name                gfs
  LV UUID                pqDoZQ-jnXZ-y0Zb-kl2s-53pP-TYWb-pzd1g3
  LV Write Access        read/write
  LV Status              NOT available
  LV Size                260.82 GB
  Current LE             66771
  Segments               1
  Allocation             inherit
  Read ahead sectors     0

  --- Logical volume ---
  LV Name                /dev/gfs/gfs1
  VG Name                gfs
  LV UUID                8MYJp6-3K1l-HtTi-qwbK-VT2p-WgO8-C98BSl
  LV Write Access        read/write
  LV Status              NOT available
  LV Size                260.82 GB
  Current LE             66771
  Segments               1
  Allocation             inherit
  Read ahead sectors     0

  --- Logical volume ---
  LV Name                /dev/gfs/gfs2
  VG Name                gfs
  LV UUID                F7SWs4-I7qj-a5Ma-no8Y-p8ir-gCEn-Z2F8EM
  LV Write Access        read/write
  LV Status              NOT available
  LV Size                260.82 GB
  Current LE             66771
  Segments               2
  Allocation             inherit
  Read ahead sectors     0

  --- Logical volume ---
  LV Name                /dev/gfs/gfs3
  VG Name                gfs
  LV UUID                4jI7ey-jWUz-KBnZ-7xfQ-xPZT-nir9-s5gQBa
  LV Write Access        read/write
  LV Status              NOT available
  LV Size                260.82 GB
  Current LE             66771
  Segments               1
  Allocation             inherit
  Read ahead sectors     0

  --- Logical volume ---
  LV Name                /dev/gfs/gfs4
  VG Name                gfs
  LV UUID                uo4iyn-vALS-DQ6j-WM4m-8nGV-vtcR-6WTM9e
  LV Write Access        read/write
  LV Status              NOT available
  LV Size                260.84 GB
  Current LE             66774
  Segments               1
  Allocation             inherit
  Read ahead sectors     0


Node WITH dm entries:
[root@tank-04 ~]# cat /proc/partitions
major minor  #blocks  name

   3     0   39082680 hda
   3     1     104391 hda1
   3     2   38973690 hda2
   8     0 1367490560 sda
   8     1  683742464 sda1
   8     2  683742465 sda2
 253     0    1048576 dm-0
 253     1   37912576 dm-1
 253     2  273494016 dm-2
 253     3  273494016 dm-3
 253     4  273494016 dm-4
 253     5  273494016 dm-5
 253     6  273506304 dm-6
[root@tank-04 ~]# dmsetup ls
gfs-gfs1        (253, 3)
gfs-gfs0        (253, 2)
VolGroup00-LogVol01     (253, 0)
gfs-gfs4        (253, 6)
VolGroup00-LogVol00     (253, 1)
gfs-gfs3        (253, 5)
gfs-gfs2        (253, 4)
[root@tank-04 ~]# lvdisplay
[HANG]


Both of these nodes are in the same cluster and connected to the same
storage. The only difference is one has a lpfc and the other a qla2300.

Reopen if it happens again.

(Raise a separate bug for the new problem)