Checked in some code to -rSTABLE for people to have a go with. I have tested it
briefly and it's fairly innocuous (I think).

You'll need Udev running with the rules file shown below and also build the dlm
library with ./confgure --have_udev.

Checking in dlm-kernel/src/device.c;
/cvs/cluster/cluster/dlm-kernel/src/device.c,v  <--  device.c
new revision: 1.24.2.1.4.1.2.6; previous revision: 1.24.2.1.4.1.2.5
done
Checking in dlm-kernel/src/dlm_device.h;
/cvs/cluster/cluster/dlm-kernel/src/dlm_device.h,v  <--  dlm_device.h
new revision: 1.4.8.1; previous revision: 1.4
done
Checking in dlm/configure;
/cvs/cluster/cluster/dlm/configure,v  <--  configure
new revision: 1.4.8.1; previous revision: 1.4
done
Checking in dlm/lib/libdlm.c;
/cvs/cluster/cluster/dlm/lib/libdlm.c,v  <--  libdlm.c
new revision: 1.15.2.2.6.1; previous revision: 1.15.2.2
done
Checking in dlm/make/defines.mk.input;
/cvs/cluster/cluster/dlm/make/defines.mk.input,v  <--  defines.mk.input
new revision: 1.1.8.1; previous revision: 1.1
done

------------------------------
udev rules file:

KERNEL=="dlm-control", NAME="misc/dlm-control"
KERNEL=="dlm_default", NAME="misc/dlm_default" MODE="0666"
KERNEL=="dlm_*", NAME="misc/%k" MODE="0660"

Here's a udev rules file that actually works on RHEL4. put it in
/etc/udev/rules.d/51-dlm.rules

KERNEL="dlm-control", NAME="misc/dlm-control", MODE="0666"
KERNEL="dlm_default", NAME="misc/dlm_default", MODE="0666"
KERNEL="dlm_*", NAME="misc/%k", MODE="0660"

and checked into RHEL4 for U4

Checking in dlm/lib/51-udev-dlm.rules;
/cvs/cluster/cluster/dlm/lib/Attic/51-udev-dlm.rules,v  <--  51-udev-dlm.rules
new revision: 1.1.2.1; previous revision: 1.1
done
Checking in dlm/lib/libdlm.c;
/cvs/cluster/cluster/dlm/lib/libdlm.c,v  <--  libdlm.c
new revision: 1.15.2.3; previous revision: 1.15.2.2
done
Checking in dlm-kernel/src/device.c;
/cvs/cluster/cluster/dlm-kernel/src/device.c,v  <--  device.c
new revision: 1.24.2.6; previous revision: 1.24.2.5
done
Checking in dlm-kernel/src/dlm_device.h;
/cvs/cluster/cluster/dlm-kernel/src/dlm_device.h,v  <--  dlm_device.h
new revision: 1.4.2.1; previous revision: 1.4
done

Tried this, but problem remains for normal users.
The /dev/misc directory is created with rx permission for other users, and if 
this is changed manually to include write-permission, the problem is solved.

Normal users don't need to be able to write into /dev/misc - that's Udev's job.

Did you add the udev rules file into /etc/udev/rules.d ?

I did add the udev rules, and it made a difference:
  Before, the /dev/dlm* files had no rwx on any users other than root.
  After, the /dev/misc/dlm* files had r-x on others and group.

When I tested and it failed, I looked at permission on /dev/misc:
drwxr-xr-x 2 root root 60 Jan  9 12:32   /dev/misc

This made me think that pds_manager(my user in its own group also called 
pds_manager) (or who ever makes the dlm_default device) probably has no 
permission to add dlm_default to the /dev/misc directory, so I did:
  chmod a+w /dev/misc
When I re-run the test, it works.

But this is not the right solution - /dev/misc is created when the system boots 
and starts the dlm daemon and must therefore be created with other permissions, 
or something else must change to allow dlm_default to be created in /dev/misc.

drwxr-xr-x  is a perfectly correct permission for /dev/misc. It's not the user
that creates the /dev/misc/dlm* files - it's Udev. So the user only needs
permission to read from the directory. Udev runs privileged.

Users need to be able to read/write the dlm-default and dlm_control files in
there though - the udev rules file I wrote sets the permissions on those files
to 0666.

If the dlm-* files that are being created have r-x permission for users then either
a) udev is not creating the device at all, libdlm is still creating them as a
fallback or
b) the rules file is wrong or being ignored.

Be careful with the rules file, RHEL4 has an older udev and needs the last of
the rules file mentioned in this bugzilla (comment #2) or from the RHEL4 brtanch
of CVS.

The rules file in comment#1 only works on later versions of udev and also needs
a MODE=0666 adding to the dlm_contorl line. (this is fixed in STABLE CVS).

Also, make sure you have the matching libdlm. The old libdlm always creates the
files. The new libdlm will wait for a while to give udev a chance to create them.

Rebuilt with --have_udev and the problem is solved!
Thank you for your patience.

An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on the solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2006-0555.html