+++ This bug was initially created as a clone of Bug #1631847 +++

Okay, I've tried to debug this as far as I can, hopefully you can take it from here.

While trying to build texlive-base for rawhide, I noticed that it had started to fail to build, despite no code changes. Specifically, it fails to build on our 32bit architectures (i686, armv7hl), but succeeds everywhere else.

The specific failure is in a test case for omegafonts:

============================================================================
Testsuite summary for Web2C 2018
============================================================================
# TOTAL: 16
# PASS:  15
# SKIP:  0
# XFAIL: 0
# FAIL:  1
# XPASS: 0
# ERROR: 0
============================================================================
See omegafonts/test-suite.log
Please report to tex-k
============================================================================

I brought up an i686 mock build of Fedora-30-i386 and reproduced this. The test-suite.log says:
FAIL: check
===========

#! /bin/sh -vx
# $Id: check.test 45809 2017-11-15 00:36:56Z karl $
# Copyright 2017 Karl Berry <tex-live>
# Copyright 2014, 2015 Peter Breitenlohner <tex-live>
# You may freely use, modify and/or distribute this file.

test -d tests || mkdir -p tests
+ test -d tests

TEXMFCNF=$srcdir/../../kpathsea
+ TEXMFCNF=../../../../texk/web2c/omegafonts/../../kpathsea
OFMFONTS=".;./tests"
+ OFMFONTS='.;./tests'
export TEXMFCNF OFMFONTS
+ export TEXMFCNF OFMFONTS

echo && echo "*** ofm2opl check xcheck"
+ echo

+ echo '*** ofm2opl check xcheck'
*** ofm2opl check xcheck
./omfonts -ofm2opl $srcdir/tests/check tests/xcheck || exit 1
+ ./omfonts -ofm2opl ../../../../texk/web2c/omegafonts/tests/check tests/xcheck
Bad OFM file: Ligature/kern step 2 skips too far;
I made it stop.
Bad OFM file: Kern index too large.
malloc(): invalid next size (unsorted)
../../../../texk/web2c/omegafonts/check.test: line 14:  9396 Aborted                 (core dumped) ./omfonts -ofm2opl $srcdir/tests/check tests/xcheck
+ exit 1
FAIL check.test (exit status: 1)

*****

Then, I mounted /proc, rebuilt omegafonts with -ggdb3, set the ENV variables like it does during `make check`, and ran the omfonts command through gdb:

[root@localhost omegafonts]# gdb --args .libs/omfonts -ofm2opl ../../../../texk/web2c/omegafonts/tests/check tests/xcheck
GNU gdb (GDB) Fedora 8.1.90.20180727-45.fc30
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "i686-redhat-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from .libs/omfonts...done.
(gdb) run
Starting program: /builddir/build/BUILD/texlive-base-20180414/source/work/texk/web2c/omegafonts/.libs/omfonts -ofm2opl ../../../../texk/web2c/omegafonts/tests/check tests/xcheck
warning: Loadable section ".note.gnu.property" outside of ELF segments
Bad OFM file: Ligature/kern step 2 skips too far;
I made it stop.
Bad OFM file: Kern index too large.
malloc(): invalid next size (unsorted)

Program received signal SIGABRT, Aborted.
0xf7fd2079 in __kernel_vsyscall ()
(gdb) bt
#0  0xf7fd2079 in __kernel_vsyscall ()
#1  0xf7e29b36 in __libc_signal_restore_set (set=0xffffcdcc) at ../sysdeps/unix/sysv/linux/internal-signals.h:84
#2  __GI_raise (sig=6) at ../sysdeps/unix/sysv/linux/raise.c:48
#3  0xf7e13374 in __GI_abort () at abort.c:79
#4  0xf7e6e37c in __libc_message (action=<optimized out>, fmt=<optimized out>) at ../sysdeps/posix/libc_fatal.c:181
#5  0xf7e753bf in malloc_printerr (str=str@entry=0xf7f52850 "malloc(): invalid next size (unsorted)") at malloc.c:5354
#6  0xf7e7802b in _int_malloc (av=av@entry=0xf7f9f7a0 <main_arena>, bytes=bytes@entry=4) at malloc.c:3727
#7  0xf7e797dd in __GI___libc_malloc (bytes=4) at malloc.c:3041
#8  0xf7fbd9e8 in xmalloc (size=4) at ../../../texk/kpathsea/xmalloc.c:25
#9  0x56559e55 in retrieve_exten_table (table=0x565d5f20 "") at ../../../../texk/web2c/omegafonts/char_routines.c:837
#10 0x56562ce7 in ofm_read_rest () at ../../../../texk/web2c/omegafonts/parse_ofm.c:371
#11 parse_ofm (read_ovf=0) at ../../../../texk/web2c/omegafonts/parse_ofm.c:99
#12 0x565579e1 in main (argc=<optimized out>, argv=<optimized out>) at ../../../../texk/web2c/omegafonts/omfonts.c:286

*****

texk/kpathsea/xmalloc.c is very short and I do not see anything obviously incorrect in it. It looks like this:

#include <kpathsea/config.h>

void *
xmalloc (size_t size)
{
    void *new_mem = (void *)malloc(size ? size : 1);

    if (new_mem == NULL) {
        fprintf(stderr, "fatal: memory exhausted (xmalloc of %lu bytes).\n",
                (unsigned long)size);
        exit(EXIT_FAILURE);
    }

    return new_mem;
}

*****

Can you help figure out what's happening here? Let me know if there is anything else I can do.

--- Additional comment from Tom "spot" Callaway on 2018-09-21 13:33:19 EDT ---

xmalloc.c:25 is:

    void *new_mem = (void *)malloc(size ? size : 1);

--- Additional comment from Florian Weimer on 2018-09-21 13:35:23 EDT ---

Have you run the program under valgrind?  What does valgrind say?  Thanks.

--- Additional comment from Tom "spot" Callaway on 2018-09-21 13:50:53 EDT ---

[root@localhost omegafonts]# valgrind .libs/omfonts -ofm2opl ../../../../texk/web2c/omegafonts/tests/check tests/xcheck
==20225== Memcheck, a memory error detector
==20225== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==20225== Using Valgrind-3.14.0.GIT and LibVEX; rerun with -h for copyright info
==20225== Command: .libs/omfonts -ofm2opl ../../../../texk/web2c/omegafonts/tests/check tests/xcheck
==20225== 
==20225== Invalid write of size 1
==20225==    at 0x10CA60: adjust_labels (char_routines.c:695)
==20225==    by 0x115CC1: ofm_read_rest (parse_ofm.c:368)
==20225==    by 0x115CC1: parse_ofm (parse_ofm.c:99)
==20225==    by 0x10A9E0: main (omfonts.c:286)
==20225==  Address 0x4b13ecc is 0 bytes after a block of size 12 alloc'd
==20225==    at 0x4837717: calloc (vg_replace_malloc.c:752)
==20225==    by 0x48555E4: xcalloc (xcalloc.c:25)
==20225==    by 0x1137C9: retrieve_ligkern_table (ligkern_routines.c:652)
==20225==    by 0x115CB5: ofm_read_rest (parse_ofm.c:367)
==20225==    by 0x115CB5: parse_ofm (parse_ofm.c:99)
==20225==    by 0x10A9E0: main (omfonts.c:286)
==20225== 
Bad OFM file: Ligature/kern step 2 skips too far;
I made it stop.
Bad OFM file: Kern index too large.

Width index for character "41 is too large;
so I reset it to zero.

Height index for character "42 is too large;
so I reset it to zero.

Depth index for character "43 is too large;
so I reset it to zero.

Ligature/kern starting index for character "44 is too large;
so I removed it.

Italic correction index for character "45 is too large;
so I reset it to zero.

Extensible index for character "46 is too large;
so I reset it to zero.
==20225== 
==20225== HEAP SUMMARY:
==20225==     in use at exit: 437,475 bytes in 979 blocks
==20225==   total heap usage: 2,867 allocs, 1,888 frees, 891,280 bytes allocated
==20225== 
==20225== LEAK SUMMARY:
==20225==    definitely lost: 4,401 bytes in 130 blocks
==20225==    indirectly lost: 896 bytes in 45 blocks
==20225==      possibly lost: 0 bytes in 0 blocks
==20225==    still reachable: 432,178 bytes in 804 blocks
==20225==         suppressed: 0 bytes in 0 blocks
==20225== Rerun with --leak-check=full to see details of leaked memory
==20225== 
==20225== For counts of detected and suppressed errors, rerun with: -v
==20225== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)

--- Additional comment from Florian Weimer on 2018-09-21 13:54:46 EDT ---

(In reply to Tom "spot" Callaway from comment #3)
> [root@localhost omegafonts]# valgrind .libs/omfonts -ofm2opl
> ../../../../texk/web2c/omegafonts/tests/check tests/xcheck
> ==20225== Memcheck, a memory error detector
> ==20225== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
> ==20225== Using Valgrind-3.14.0.GIT and LibVEX; rerun with -h for copyright
> info
> ==20225== Command: .libs/omfonts -ofm2opl
> ../../../../texk/web2c/omegafonts/tests/check tests/xcheck
> ==20225== 
> ==20225== Invalid write of size 1
> ==20225==    at 0x10CA60: adjust_labels (char_routines.c:695)
> ==20225==    by 0x115CC1: ofm_read_rest (parse_ofm.c:368)
> ==20225==    by 0x115CC1: parse_ofm (parse_ofm.c:99)
> ==20225==    by 0x10A9E0: main (omfonts.c:286)
> ==20225==  Address 0x4b13ecc is 0 bytes after a block of size 12 alloc'd
> ==20225==    at 0x4837717: calloc (vg_replace_malloc.c:752)
> ==20225==    by 0x48555E4: xcalloc (xcalloc.c:25)
> ==20225==    by 0x1137C9: retrieve_ligkern_table (ligkern_routines.c:652)
> ==20225==    by 0x115CB5: ofm_read_rest (parse_ofm.c:367)
> ==20225==    by 0x115CB5: parse_ofm (parse_ofm.c:99)
> ==20225==    by 0x10A9E0: main (omfonts.c:286)
> ==20225== 

Thanks!

This is the kind of heap corruption the new checks in glibc malloc are supposed to catch.  It's a latent bug in omfonts.

It's only visible on 32-bit architectures because on others, the chunk size is rounded up to a larger value.