Bug 1631847

Okay, I've tried to debug this as far as I can, hopefully you can take it from here.

While trying to build texlive-base for rawhide, I noticed that it had started to fail to build, despite no code changes. Specifically, it fails to build on our 32bit architectures (i686, armv7hl), but succeeds everywhere else.

The specific failure is in a test case for omegafonts:

============================================================================
Testsuite summary for Web2C 2018
============================================================================
# TOTAL: 16
# PASS:  15
# SKIP:  0
# XFAIL: 0
# FAIL:  1
# XPASS: 0
# ERROR: 0
============================================================================
See omegafonts/test-suite.log
Please report to tex-k
============================================================================

I brought up an i686 mock build of Fedora-30-i386 and reproduced this. The test-suite.log says:
FAIL: check
===========

#! /bin/sh -vx
# $Id: check.test 45809 2017-11-15 00:36:56Z karl $
# Copyright 2017 Karl Berry <tex-live>
# Copyright 2014, 2015 Peter Breitenlohner <tex-live>
# You may freely use, modify and/or distribute this file.

test -d tests || mkdir -p tests
+ test -d tests

TEXMFCNF=$srcdir/../../kpathsea
+ TEXMFCNF=../../../../texk/web2c/omegafonts/../../kpathsea
OFMFONTS=".;./tests"
+ OFMFONTS='.;./tests'
export TEXMFCNF OFMFONTS
+ export TEXMFCNF OFMFONTS

echo && echo "*** ofm2opl check xcheck"
+ echo

+ echo '*** ofm2opl check xcheck'
*** ofm2opl check xcheck
./omfonts -ofm2opl $srcdir/tests/check tests/xcheck || exit 1
+ ./omfonts -ofm2opl ../../../../texk/web2c/omegafonts/tests/check tests/xcheck
Bad OFM file: Ligature/kern step 2 skips too far;
I made it stop.
Bad OFM file: Kern index too large.
malloc(): invalid next size (unsorted)
../../../../texk/web2c/omegafonts/check.test: line 14:  9396 Aborted                 (core dumped) ./omfonts -ofm2opl $srcdir/tests/check tests/xcheck
+ exit 1
FAIL check.test (exit status: 1)

*****

Then, I mounted /proc, rebuilt omegafonts with -ggdb3, set the ENV variables like it does during `make check`, and ran the omfonts command through gdb:

[root@localhost omegafonts]# gdb --args .libs/omfonts -ofm2opl ../../../../texk/web2c/omegafonts/tests/check tests/xcheck
GNU gdb (GDB) Fedora 8.1.90.20180727-45.fc30
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "i686-redhat-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from .libs/omfonts...done.
(gdb) run
Starting program: /builddir/build/BUILD/texlive-base-20180414/source/work/texk/web2c/omegafonts/.libs/omfonts -ofm2opl ../../../../texk/web2c/omegafonts/tests/check tests/xcheck
warning: Loadable section ".note.gnu.property" outside of ELF segments
Bad OFM file: Ligature/kern step 2 skips too far;
I made it stop.
Bad OFM file: Kern index too large.
malloc(): invalid next size (unsorted)

Program received signal SIGABRT, Aborted.
0xf7fd2079 in __kernel_vsyscall ()
(gdb) bt
#0  0xf7fd2079 in __kernel_vsyscall ()
#1  0xf7e29b36 in __libc_signal_restore_set (set=0xffffcdcc) at ../sysdeps/unix/sysv/linux/internal-signals.h:84
#2  __GI_raise (sig=6) at ../sysdeps/unix/sysv/linux/raise.c:48
#3  0xf7e13374 in __GI_abort () at abort.c:79
#4  0xf7e6e37c in __libc_message (action=<optimized out>, fmt=<optimized out>) at ../sysdeps/posix/libc_fatal.c:181
#5  0xf7e753bf in malloc_printerr (str=str@entry=0xf7f52850 "malloc(): invalid next size (unsorted)") at malloc.c:5354
#6  0xf7e7802b in _int_malloc (av=av@entry=0xf7f9f7a0 <main_arena>, bytes=bytes@entry=4) at malloc.c:3727
#7  0xf7e797dd in __GI___libc_malloc (bytes=4) at malloc.c:3041
#8  0xf7fbd9e8 in xmalloc (size=4) at ../../../texk/kpathsea/xmalloc.c:25
#9  0x56559e55 in retrieve_exten_table (table=0x565d5f20 "") at ../../../../texk/web2c/omegafonts/char_routines.c:837
#10 0x56562ce7 in ofm_read_rest () at ../../../../texk/web2c/omegafonts/parse_ofm.c:371
#11 parse_ofm (read_ovf=0) at ../../../../texk/web2c/omegafonts/parse_ofm.c:99
#12 0x565579e1 in main (argc=<optimized out>, argv=<optimized out>) at ../../../../texk/web2c/omegafonts/omfonts.c:286

*****

texk/kpathsea/xmalloc.c is very short and I do not see anything obviously incorrect in it. It looks like this:

#include <kpathsea/config.h>

void *
xmalloc (size_t size)
{
    void *new_mem = (void *)malloc(size ? size : 1);

    if (new_mem == NULL) {
        fprintf(stderr, "fatal: memory exhausted (xmalloc of %lu bytes).\n",
                (unsigned long)size);
        exit(EXIT_FAILURE);
    }

    return new_mem;
}

*****

Can you help figure out what's happening here? Let me know if there is anything else I can do.