Description of problem:
Reported by Solar Designer.

This issue was fixed during the 2.0/2.3 kernel days:
http://marc.info/?l=bugtraq&m=93728282528791

"(2.0.38) (2.2.12) (*)
While investigating the problem mentioned above, as well as porting Pavel Kankovsky's fd 0-2 fix to Linux 2.2, I've noticed yet another problem with execve(2), which has a similar impact.  The problem is that all arguments are first counted, and their lengths measured, and only after that the results are checked against the limit (32 pages, or 128 KB on x86).  Thus, it is possible to make execve(2) spend a significant amount of CPU cycles in the kernel, with the big kernel lock obtained.  In terms of real time, I was able to get 25 minutes for one execve(2) call on Alpha (the 64-bit address space helps), and
several seconds on modern x86 boxes.  The latter can still be halted to death by repeating the call in a loop, and in a few processes.

Fixing this required a trivial modification to the argument counting function, and a switch to strnlen_user() for the strings (on 2.2). The latter is an architecture-specific assembly function.  My patch only fixes it on x86 and Alpha, and 2.2.13 will do that for some more architectures (but maybe not all supported by Linux 2.2, yet).  Linux 2.0 didn't have a strlen_user(), and thus used a simple loop in this place, which had the same vulnerability (but not in my 2.0.38 patch, of course).

Note that RLIMIT_AS can be used as a workaround for this problem, and you should probably be using it for other reasons, anyway."

But it was reported to be reintroduced:
http://seclists.org/oss-sec/2010/q3/247

"Now it appears that, besides the issue that started this thread, the same problem I mentioned above got re-introduced.  We still have strnlen_user() and the "max" argument to count(), but we no longer have hard limits for "max".  Someone set MAX_ARG_STRINGS to 0x7FFFFFFF, and this is just too much.  MAX_ARG_STRLEN is set to 32 pages, and these two combined allow a userspace program to make the kernel loop for days.

So I think that we should re-introduce some artificial limit(s), maybe adjustable by root (by the host system's real root only when container virtualization is involved).  Maybe we should lower MAX_ARG_STRINGS and/or maybe we should limit the portion of stack space usable for argv to, say, 0.75 GB (or even less)."

He suggested:
  -               if (size > rlim[RLIMIT_STACK].rlim_cur / 4) {
  +               if (size > rlim[RLIMIT_STACK].rlim_cur / 4 ||
  +                   size > MAX_ARG_STACK) {

  -#define MAX_ARG_STRINGS 0x7FFFFFFF
  +#define MAX_ARG_STACK (0xC0000000UL / 4)
  +#define MAX_ARG_STRINGS (MAX_ARG_STACK / 5)

but may not be a sufficient solution.