Bug 1007678 (CVE-2013-4359)

ProFTPd default installation comes with mod_sftp and mod_sftp_pam activated, which initiates this flaw.
The bug is useful to trigger a large heap allocation and exhaust all available system memory of the underlying operating system.

If we look at the source code of kbdint.c: 
https://github.com/proftpd/proftpd-test-convert/blob/master/contrib/mod_sftp/kbdint.c

We'd find the lines:

resp_count = sftp_msg_read_int(pkt->pool, &buf, &buflen);

  list = make_array(p, resp_count, sizeof(char *));
  for (i = 0; i < resp_count; i++) {
    char *resp;

    resp = sftp_msg_read_string(pkt->pool, &buf, &buflen);
    *((char **) push_array(list)) = pstrdup(p, sftp_utf8_decode_str(p, resp));
  }

  *count = (unsigned int) resp_count;
  *responses = ((const char **) list->elts);
  return 0;
}


First line will read the kbdint response count resp_count  which is an unsigned integer with a size of 32 bits from the client during an SSH kbdint userauth info response client request.
This value is used to allocate a buffer with the size user_supplied_uint32_value multiplied by the size of a char pointer being 32bits or 64bits depending on the platform.
We can see that no size check is performed before the request is sent to pool allocator called by make_array in 2nd line.
The pool allocator can be tricked to handle negative allocation sizes if resp_count is large enough.
There is a size check of the response count value but it’s done after this function returns.
The DoS condition can be triggered by sending an int32 value for resp_count that is slightly below the available memory of the target system and repeating the request.

References:

https://bugs.gentoo.org/show_bug.cgi?id=484614
http://kingcope.wordpress.com/2013/09/11/proftpd-mod_sftpmod_sftp_pam-invalid-pool-allocation-in-kbdint-authentication/