There is a minor heap overflow resulting in heap corruption in
check_idx() at mkind.c:410. It is definitely not a security issue,
because it doesn't allow an attacker to inject arbitrary data on stack.

But it might be unrelated, because it requires the filename to
be at least 252 characters long, which is not the case in
the above description.

Let's call makeindex this way:
$ makeindex `perl -e 'print "x" x 255'`

Here the filename length gets checked against STRING_MAX
which is 256 characters:

386     if (i < STRING_MAX)
387         base[i] = NUL;
388     else
389         FATAL2("Index file name %s too long (max %d).\n",
390                base, STRING_MAX);

(gdb) print STRING_MAX
$4 = 256

Then the same amount of data gets allocated to idx_fn and the file name
extended by 4 characters gets copied there:

408             if ((idx_fn = (char *) malloc(STRING_MAX)) == NULL)
409                 FATAL("Not enough core...abort.\n", "");
410             sprintf(idx_fn, "%s%s", base, INDEX_IDX);
	
(gdb) print INDEX_IDX
$5 = ".idx"
(gdb) print strlen(base)
$6 = 255

Shoul be fixed by adjusting the conditional on line 386 accordingly I guess.

The previous bug was really a different one.
The actual problem here is that makeindex does

516             strcpy(sty_fn, fn);

in open_sty (fn) at mkind.c. fn gets passed without any check from
a program agrument in main():

150                         /* style file */
151                     case 's':
152                         argc--;
153                         if (argc <= 0)
154                             FATAL("Expected -s <stylefile>\n","");
155                         open_sty(*++argv);
156                         sty_given = TRUE;
157                         break;

however, sty_fn is is only 72 characters wide. See mkindex.h:

 56 char    sty_fn[LINE_MAX];

(gdb) print LINE_MAX
$2 = 72
(gdb)

This allows a malicious attacker to owerwrite data that follows the
sty_fn[] buffer almost arbitrarily. This might lead to overwrite of
arbitrary files if pathnames were stored there or arbitrary code
execution in case there is a pointer to function there.

Yet another buffer overflow at mkind.c:182:

strcpy(pageno, *++argv);

where the argument string is immediatelly assigned to a global array.

Created attachment 147214 [details]
Proposed patch to fix the makeindex issues.

tetex-3.0-34.fc6 has been pushed for fc6, which should resolve this issue.  If these problems are still present in this version, then please make note of it in this bug report.

Just tested it again with tetex-3.0-39.fc7.x86_64 on F7. The buffer overflow is
gone. Thanks for this. Although, the error message is a bit confusing:

10008:$ makeindex -s
/home/mr/perforce/tools/dblatex-0.2.3/share/dblatex/latex/scripts/doc.ist
Style file
/home/mr/perforce/tools/dblatex-0.2.3/share/dblatex/latex/scripts/doc.ist too long.

It says that the style file is too long, but actually it's the file name that is
too long. The same file works after copying to /tmp:

10010:$ cp
/home/mr/perforce/tools/dblatex-0.2.3/share/dblatex/latex/scripts/doc.ist /tmp
mr@ash:~
10011:$ makeindex -s /tmp/doc.ist 
Scanning style file /tmp/doc.ist....done (4 attributes redefined, 0 ignored).
This is makeindex, version 2.14 [02-Oct-2002] (kpathsea + Thai support).
Scanning input file stdin...

Still, I wonder why the file name length is so much limited at all (especially
since the path above is not really long).

Fedora apologizes that these issues have not been resolved yet. We're
sorry it's taken so long for your bug to be properly triaged and acted
on. We appreciate the time you took to report this issue and want to
make sure no important bugs slip through the cracks.

If you're currently running a version of Fedora Core between 1 and 6,
please note that Fedora no longer maintains these releases. We strongly
encourage you to upgrade to a current Fedora release. In order to
refocus our efforts as a project we are flagging all of the open bugs
for releases which are no longer maintained and closing them.
http://fedoraproject.org/wiki/LifeCycle/EOL

If this bug is still open against Fedora Core 1 through 6, thirty days
from now, it will be closed 'WONTFIX'. If you can reporduce this bug in
the latest Fedora version, please change to the respective version. If
you are unable to do this, please add a comment to this bug requesting
the change.

Thanks for your help, and we apologize again that we haven't handled
these issues to this point.

The process we are following is outlined here:
http://fedoraproject.org/wiki/BugZappers/F9CleanUp

We will be following the process here:
http://fedoraproject.org/wiki/BugZappers/HouseKeeping to ensure this
doesn't happen again.

And if you'd like to join the bug triage team to help make things
better, check out http://fedoraproject.org/wiki/BugZappers

Comment #7 is also valid for Fedora 8 with tetex-3.0-44.8.fc8.x86_64.

thanks for the update!

Ok, I updated the CVE-2005-0650 patch to increase the file name size limit to
STRING_MAX (256) from LINE_MAX (72) and the error message. The fix for it should
occur in the next update.

tetex-3.0-44.9.fc8 has been submitted as an update for Fedora 8

tetex-3.0-44.9.fc8 has been pushed to the Fedora 8 testing repository.  If problems still persist, please make note of it in this bug report.
 If you want to test the update, you can install it with 
 su -c 'yum --enablerepo=updates-testing update tetex'.  You can provide feedback for this update here: http://admin.fedoraproject.org/updates/F8/FEDORA-2008-3158

This message is a reminder that Fedora 8 is nearing its end of life.
Approximately 30 (thirty) days from now Fedora will stop maintaining
and issuing updates for Fedora 8.  It is Fedora's policy to close all
bug reports from releases that are no longer maintained.  At that time
this bug will be closed as WONTFIX if it remains open with a Fedora 
'version' of '8'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version prior to Fedora 8's end of life.

Bug Reporter: Thank you for reporting this issue and we are sorry that 
we may not be able to fix it before Fedora 8 is end of life.  If you 
would still like to see this bug fixed and are able to reproduce it 
against a later version of Fedora please change the 'version' of this 
bug to the applicable version.  If you are unable to change the version, 
please add a comment here and someone will do it for you.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events.  Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

The process we are following is described here: 
http://fedoraproject.org/wiki/BugZappers/HouseKeeping

Fedora 8 changed to end-of-life (EOL) status on 2009-01-07. Fedora 8 is 
no longer maintained, which means that it will not receive any further 
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of 
Fedora please feel free to reopen this bug against that version.

Thank you for reporting this bug and we are sorry it could not be fixed.