The CUPS "pdftops" filter converts Portable Document Format (PDF) files into PostScript. It is based upon Xpdf and the CUPS imaging library.

Multiple integer overflows, leading to heap-based buffer overflows, were
found in the CUPS "pdftops" filter. An attacker could create a malicious
PDF file that would cause "pdftops" to crash or, potentially, execute
arbitrary code as the "lp" user if the file was printed.

This issue has been addressed in following products:

  Red Hat Enterprise Linux 3
  Red Hat Enterprise Linux 4

Via RHSA-2009:1083 https://rhn.redhat.com/errata/RHSA-2009-1083.html

Patch for this issue fixes all cases where gmalloc is called with argument consisting of multiplication of multiple values (such as gmalloc(a*b)).  If one of the values is read from input file and is not previously checked, multiplication operation may overflow resulting in the insufficient memory allocation.  All such calls were replaced with use of gmallocn (i.e. gmallocn(a,b)).  Most of those changes were done in the xpdf code base in version 3.

Part of the patch is applicable and was applied to CUPS version shipped in Red Hat Enterprise Linux 5.  However, none of those changes has security implications or may not even be triggerable.  Detailed analysis of the cases addressed in RHEL5 patch:


+++ cups-1.3.7/pdftops/Decrypt.cxx
-  buf = (Guchar *)gmalloc(72 + fileID->getLength());
+  buf = (Guchar *)gmallocCn(72, 1, fileID->getLength());

This is basically "strlen(something-already-in-memory) + very-small-constant".  With such a small constant, it's very unlikely large enough input can be placed into memory and actually trigger an overflow, as it would require input of size very close to the addressable memory size limit.


+++ cups-1.3.7/pdftops/FoFiTrueType.cxx
-  tableDir = (char *)gmalloc(12 + nNewTables * 16);
+  tableDir = (char *)gmallocCn(12, nNewTables, 16);

Few lines above, there is:

  newTables = (TrueTypeTable *)gmallocn(nNewTables, sizeof(TrueTypeTable));

where TrueTypeTable is:

  struct TrueTypeTable {
    Guint tag;
    Guint checksum;
    int offset;
    int origOffset;
    int len;
  };

so 20 bytes on all our platforms.  If nNewTables is large enough to int
overflow on "12 + nNewTables * 16", it is large enough to overflow in the
mentioned gmallocn call above and cause abort there.


+++ cups-1.3.7/pdftops/gmem.c
-  s1 = (char *)gmalloc(strlen(s) + 1);
+  s1 = (char *)gmallocCn(1, 1, strlen(s));

"strlen(something-already-in-memory) + very-small-constant" case again.


+++ cups-1.3.7/pdftops/JBIG2Stream.cxx
-  data = (Guchar *)gmalloc(h * line + 1);
+  data = (Guchar *)gmallocCn(1, h, line);

-  data = (Guchar *)gmalloc(h * line + 1);
+  data = (Guchar *)gmallocCn(1, h, line);

There is already an int overflow check right above those two gmalloc calls:

  if (w <= 0 || h <= 0 || line <= 0 || h >= (INT_MAX - 1) / line) {
    // force a call to gmalloc(-1), which will throw an exception
    h = -1;
    line = 2;
  }

This check was added as part of the JBIG issues security patch.


+++ cups-1.3.7/pdftops/PSOutputDev.cxx
-  lineBuf = (Guchar *)gmalloc(4 * width);
+  lineBuf = (Guchar *)gmallocn(4, width);

Affected function can only be reached, when psLevel1Sep PostScript language level is used. Looking into pdftops, it defaults to psLevel2 and can switch to psLevel1 if PPD says so.

Those unsafe gmalloc uses are not specific to xpdf fork used by CUPS, similar problems affects other Xpdf versions / forks too.

These issues were already addressed in CUPS in Red Hat Enterprise Linux 3 and 4 (see comment #12) and none of them has any security impact on Red Hat Enterprise Linux 5 (see comment #13).

Patch hardening gmalloc use was applied to poppler packages in Red Hat Enterprise Linux 5 via:
  https://rhn.redhat.com/errata/RHSA-2009-0480.html

Other packages including xpdf code base will have this preventive patch applied in the future updates.

This issue has been addressed in following products:

  Red Hat Enterprise Linux 5

Via RHSA-2009:1502 https://rhn.redhat.com/errata/RHSA-2009-1502.html

This issue has been addressed in following products:

  Red Hat Enterprise Linux 4

Via RHSA-2009:1501 https://rhn.redhat.com/errata/RHSA-2009-1501.html

This issue has been addressed in following products:

  Red Hat Enterprise Linux 3

Via RHSA-2009:1500 https://rhn.redhat.com/errata/RHSA-2009-1500.html

This issue has been addressed in following products:

  Red Hat Enterprise Linux 4

Via RHSA-2009:1503 https://rhn.redhat.com/errata/RHSA-2009-1503.html

This issue has been addressed in following products:

  Red Hat Enterprise Linux 4

Via RHSA-2009:1512 https://rhn.redhat.com/errata/RHSA-2009-1512.html

Note: CVE-2009-3605 is duplicate for the same unsafe gmalloc use in xpdf / poppler code.

This issue has been addressed in following products:

  Red Hat Enterprise Linux 4

Via RHSA-2010:0399 https://rhn.redhat.com/errata/RHSA-2010-0399.html

This issue has been addressed in following products:

  Red Hat Enterprise Linux 5

Via RHSA-2010:0400 https://rhn.redhat.com/errata/RHSA-2010-0400.html

This issue has been addressed in following products:

  Red Hat Enterprise Linux 3

Via RHSA-2010:0401 https://rhn.redhat.com/errata/RHSA-2010-0401.html