Upstream removed the test in:

commit cf2422dc467f1e10a8c20cd362f7b3b296c24529
Author: Karel Miko <karel.miko>
Date:   Wed Oct 24 22:27:15 2018 +0200

    exclude wycheproof tests (too big) from dist tarball (via MANIFEST.SKIP)

Rebuilding fails with the same error even with Perl 5.30 in a standard Rawhide.

F31 is clear. I suspect an upgrade of libtommath to 1.2.0 in fc33.

I confirm this is triggered with upgrading libtommath from 1.1.0-1.fc32 to 1.2.0-1.fc33.

libtommath bisected to this breaking commint:

commit abdb03340255ff6759f2da2d400c50e4601156d6 (HEAD, refs/bisect/bad)
Author: czurnieden <czurnieden>
Date:   Wed Sep 25 00:29:19 2019 +0200

    Refactored functions to read and write binaries and added "maxlen"

It causes t/wycheproof.t test #536 to hang and then segfault. Before the commit:

$ LD_PRELOAD=/tmp/libtommath/.libs/libtommath.so   perl -Iblib/{lib,arch} t/wycheproof.t
[...]
ok 530 - type=DSAVer/SHA224 tcId=32 comment='uint64 overflow in length' expected-result=invalid
ok 531 - type=DSAVer/SHA224 tcId=33 comment='uint64 overflow in length' expected-result=invalid
ok 532 - type=DSAVer/SHA224 tcId=34 comment='length = 2**31 - 1' expected-result=invalid
ok 533 - type=DSAVer/SHA224 tcId=35 comment='length = 2**31 - 1' expected-result=invalid
ok 534 - type=DSAVer/SHA224 tcId=36 comment='length = 2**31 - 1' expected-result=invalid
ok 535 - type=DSAVer/SHA224 tcId=37 comment='length = 2**32 - 1' expected-result=invalid
ok 536 - type=DSAVer/SHA224 tcId=38 comment='length = 2**32 - 1' expected-result=invalid ←
ok 537 - type=DSAVer/SHA224 tcId=39 comment='length = 2**32 - 1' expected-result=invalid
[...]

After the commit:

ok 530 - type=DSAVer/SHA224 tcId=32 comment='uint64 overflow in length' expected-result=invalid
ok 531 - type=DSAVer/SHA224 tcId=33 comment='uint64 overflow in length' expected-result=invalid
ok 532 - type=DSAVer/SHA224 tcId=34 comment='length = 2**31 - 1' expected-result=invalid
ok 533 - type=DSAVer/SHA224 tcId=35 comment='length = 2**31 - 1' expected-result=invalid
ok 534 - type=DSAVer/SHA224 tcId=36 comment='length = 2**31 - 1' expected-result=invalid
ok 535 - type=DSAVer/SHA224 tcId=37 comment='length = 2**32 - 1' expected-result=invalid
Segmentation fault (core dumped)

GDB slows it down up to an unusable pace. /lib/libSegFault.so shows:

*** Segmentation fault
Register dump:

 EAX: 02809000   EBX: b7f1d000   ECX: 00000008   EDX: 02809001
 ESI: ffffffff   EDI: b74dba40   EBP: 027672f8   ESP: bfd3f3a0

 EIP: b7f06d71   EFLAGS: 00010246   

 CS: 0073   DS: 007b   ES: 007b   FS: 0000   GS: 0033   SS: 007b

 Trap: 0000000e   Error: 00000004   OldMask: 00000000
 ESP/signal: bfd3f3a0   CR2: 02809000

 FPUCW: ffff037f   FPUSW: ffff0020   TAG: ffffffff
 IPOFF: b7c57b44   CSSEL: 0000   DATAOFF: 00000000   DATASEL: 0000

 ST(0) 0000 0000000000000000   ST(1) 0000 0000000000000000
 ST(2) 0000 0000000000000000   ST(3) 0000 0000000000000000
 ST(4) 0000 0000000000000000   ST(5) 0000 0000000000000000
 ST(6) 0000 9bc2f43d2b140000   ST(7) 0000 0000000000000000

Backtrace:
/tmp/libtommath/.libs/libtommath.so(mp_from_ubin+0x82)[0xb7f06d71]
/tmp/libtommath/.libs/libtommath.so(mp_read_unsigned_bin+0x25)[0xb7f03f7c]
/lib/libtomcrypt.so.1(+0x61901)[0xb744e901]
/lib/libtomcrypt.so.1(der_decode_integer+0x74)[0xb745da44]
/lib/libtomcrypt.so.1(der_decode_sequence_ex+0x884)[0xb7460274]
/lib/libtomcrypt.so.1(dsa_verify_hash+0xed)[0xb7467ead]
blib/arch/auto/CryptX/CryptX.so(+0x1c585)[0xb7500585]
/lib/libperl.so.5.30(Perl_pp_entersub+0x210)[0xb7c5ee00]
/lib/libperl.so.5.30(Perl_runops_standard+0x3d)[0xb7c5500d]
/lib/libperl.so.5.30(perl_run+0x34b)[0xb7bc2d7b]
perl(+0x1398)[0x494398]
/lib/libc.so.6(__libc_start_main+0xf9)[0xb79d4fa9]
perl(+0x13e5)[0x4943e5]

"ok 536 - type=DSAVer/SHA224 tcId=38 comment='length = 2**32 - 1' expected-result=invalid" corresponds to this JSON input of the Perl test:

        {
          "comment" : "length = 2**32 - 1",
          "message" : "54657374",
          "name" : "RsaSignatureTestVector",
          "padding" : "30353084ffffffff060960864801650304020105000420532eaabd9574880dbf76b9b8cc00832c20a6ec113d682299550d7a6e0f345e25",
          "result" : "invalid",
          "sig" : "31afd9a0d827755352b16de04de42e98a8c72f08919ed475530a00c762b8a03bde22634dd856a7eede4b4947d780cb3efe55775e16d7f46f209dbcb5569b2d9469cc271aa850f74960f7c741928055925349821e32e1e0fe5a040010a39a4b6a343f7f35c204106b3617e528a99dcaea8a93766adcfe7be31cdb98f7f7f14669",
          "tcId" : 38
        },

I will try to minimize it. It could be a bad usage of libtommath API from the Perl side.

A sample backtrace when the test runs for a very long time, some time before it crashes:

(gdb) bt
#0  0xb7fb993b in mp_mul_2d (a=0xa6b080, b=8, c=0xa6b080) at bn_mp_mul_2d.c:58
#1  0xb7fb6d52 in mp_from_ubin (a=0xa6b080, b=0xab907e "", size=4294845784) at bn_mp_from_ubin.c:23
#2  0xb7fb3f7c in mp_read_unsigned_bin (a=0xa6b080, 
    b=0xa9b5d8 "\036A\264y\255Wi\005\271`\376\024\352ۑ\260\314\363HCڹ\026\027;\270\311\315\002\035", c=-1) at bn_deprecated.c:243
#3  0xb74fd8a1 in unsigned_read (a=0xa6b080, 
    b=0xa9b5d8 "\036A\264y\255Wi\005\271`\376\024\352ۑ\260\314\363HCڹ\026\027;\270\311\315\002\035", len=4294967295)
    at src/math/ltm_desc.c:201
#4  0xb750d0f5 in der_decode_integer (
    in=0xa9b5d2 "\002\204\377\377\377\377\036A\264y\255Wi\005\271`\376\024\352ۑ\260\314\363HCڹ\026\027;\270\311\315\002\035", inlen=65, 
    num=0xa6b080) at src/pk/asn1/der/integer/der_decode_integer.c:57
#5  0xb750f9a3 in der_decode_sequence_ex (
    in=0xa9b5d0 "0A\002\204\377\377\377\377\036A\264y\255Wi\005\271`\376\024\352ۑ\260\314\363HCڹ\026\027;\270\311\315\002\035", 
    inlen=<optimized out>, list=0xbffff22c, outlen=2, ordered=1) at src/pk/asn1/der/sequence/der_decode_sequence_ex.c:111
#6  0xb75176ad in dsa_verify_hash (
    sig=0xa9b5d0 "0A\002\204\377\377\377\377\036A\264y\255Wi\005\271`\376\024\352ۑ\260\314\363HCڹ\026\027;\270\311\315\002\035", 
    siglen=67, hash=0xacb870 "AI\332\030\252\213\374+\036\070,l&Um\001\251,&\033d6\332\325\343\276", <incomplete sequence \314>, 
    hashlen=28, stat=0xbffff2d0, key=0xa99f6c) at src/pk/dsa/dsa_verify_hash.c:114
#7  0xb75b05b5 in XS_Crypt__PK__DSA__verify (my_perl=0x4051a0, cv=0x8d86b4) at ./inc/CryptX_PK_DSA.xs.inc:334
#8  0xb7d0ee00 in Perl_pp_entersub () from /lib/libperl.so.5.30
#9  0xb7d0500d in Perl_runops_standard () from /lib/libperl.so.5.30
#10 0xb7c72d7b in perl_run () from /lib/libperl.so.5.30
#11 0x00401398 in main ()

The test exhibits 2**32 - 1 (4294967295) length. Here we can see the big number in mp_from_ubin(..,size=4294845784)
and a wrap in bn_deprecated(...,c=-1). Then is a libtomcrypt library with again a big number unsigned_read(...,len=4294967295).

When I read the commit, I can see:

+#ifdef BN_MP_READ_UNSIGNED_BIN_C
+mp_err mp_read_unsigned_bin(mp_int *a, const unsigned char *b, int c)
+{
+   return mp_from_ubin(a, b, (size_t) c);
+}
+#endif

That's suspicious, then then:

@@ -76,7 +76,8 @@ mp_err s_mp_prime_random_ex(mp_int *a, int t, int size, int flags, private_mp_pr
       tmp[bsize-1]             |= maskOR_lsb;
 
       /* read it in */
-      if ((err = mp_read_unsigned_bin(a, tmp, bsize)) != MP_OKAY) {
+      /* TODO: casting only for now until all lengths have been changed to the type "size_t"*/
+      if ((err = mp_from_ubin(a, tmp, (size_t)bsize)) != MP_OKAY) {
          goto error;
       }

Where a comment admits that it's indeed wrong.

I conclude it's a bug libtommath library. I will probably disable the test as it tests as it exhibits libtomcrypt/libtommath internals and that's not interested from perl-CryptX point of view. But I will report it against libtommath to know about that.

I removed the test.