When anaconda sets up /mnt/sysimage before installing RPMs, it doesn't mount devpts on /mnt/sysimage/dev/pts, so programs like expect that require pty's can't be used in RPM scripts. It should do the mount.
I believe this should be fixed in F11, which means it'll be fixed in RHEL6 as well. If you require this fix in an update release of RHEL5, please talk to your support representative who will raise this through the appropriate channels so we can get a fix scheduled.