mkstemp("/var/tmp/mkstempXXXXXX");
This is of course going to fail, no matter whether there are enough file
descriptors or not.
Unless -fwritable-strings (which is deprecated), "/var/tmp/mkstempXXXXXX"
is read-only, but:
"The mkstemp() function shall replace the contents of the string pointed to by
template by a unique filename"
The pointer passed to mkstemp must point into a writable char array.
So
char filename[] = "/var/tmp/mkstempXXXXXX";
mkstemp (filename);
is ok, but
mkstemp ("/var/tmp/mkstempXXXXXX")
is not.

As for open, I haven't tried to reproduce it, but if open really creates
a file even when it returns -1, that would be a POSIX violation.
POSIX on open says:
"No files shall be created or modified if the function returns -1."

If ulimit -n is reached, then open with O_CREAT in my testing doesn't create
the file, but the testcase is about reaching the system wide limit of file
descriptors.

Created attachment 118155 [details]
Proposed patch

The problem is the ordering of creating the entry in the directory and
allocating the file struct to reference the file.  The file struct was
getting allocated second, but should be allocated before the directory
entry is created.  Thus, if the allocation fails, then the error can be
returned without creating the directory entry.

Created attachment 119023 [details]
Proposed patch

A fix for this problem has just been committed to the RHEL3 U7
patch pool this evening (in kernel version 2.4.21-37.4.EL).

An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on the solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHSA-2006-0144.html