There are actually 2 issues here:

The first one (which is causing the error in the description) is that Heimdal client is using a 1024-bit Diffie-Hellman key, which is rejected by MIT KDC[1] because it requires a minimum of 2048 bits by default[2].

But if we reduce the minimum size of the DH key to 1024 bits in the configuration of the KDC ("[kdcdefaults] pkinit_dh_min_bits = 1024"), then a different error occurs:

  PKINIT: no freshness token received from alice.COM
  AS_REQ (2 etypes {aes256-cts-hmac-sha384-192(20), aes256-cts-hmac-sha1-96(18)}) 10.0.139.148: KDC_RETURN_PADATA: alice.COM for krbtgt/TEST.REDHAT.COM.COM, Cannot allocate memory

But the memory allocation error is wrong. The actual error comes from the fact OpenSSL fails to generate the DH key because the q prime has a size of 1023 bits, which is invalid:

#0  default_mdname (N=1023) at crypto/ffc/ffc_params_generate.c:448
#1  ossl_ffc_params_FIPS186_2_gen_verify.constprop.0 (libctx=libctx@entry=0x0, params=params@entry=0x7ffd42c8bf40, mode=mode@entry=0, L=L@entry=1024, N=1023, res=res@entry=0x7ffd42c8bf3c, cb=0x0, type=<optimized out>) at crypto/ffc/ffc_params_generate.c:836
#2  0x00007fb86c2bf5a3 in ossl_ffc_params_FIPS186_2_validate.constprop.0.isra.0 (libctx=0x0, params=0x7ffd42c8bf40, res=0x7ffd42c8bf3c, cb=0x0, type=<optimized out>) at crypto/ffc/ffc_params_validate.c:84
#3  0x00007fb86c1db7e6 in ossl_ffc_params_simple_validate (libctx=0x0, params=0x560b5b820e28, paramstype=1, res=0x7ffd42c8bf3c) at crypto/ffc/ffc_params_validate.c:116
#4  0x00007fb86c15c02f in generate_key (dh=0x560b5b820e20) at crypto/dh/dh_key.c:337
#5  generate_key (dh=0x560b5b820e20) at crypto/dh/dh_key.c:256
#6  0x00007fb86c29fae0 in dh_gen (genctx=0x560b5b92f8a0, osslcb=<optimized out>, cbarg=0x560b5b92f920) at providers/implementations/keymgmt/dh_kmgmt.c:763
#7  0x00007fb86c1d3b99 in evp_keymgmt_gen (keymgmt=<optimized out>, keymgmt=<optimized out>, cb=0x7fb86c1d1f50 <ossl_callback_to_pkey_gencb>, cbarg=0x560b5b92f920, genctx=<optimized out>) at crypto/evp/keymgmt_meth.c:375
#8  evp_keymgmt_util_gen (cb=0x7fb86c1d1f50 <ossl_callback_to_pkey_gencb>, cbarg=0x560b5b92f920, genctx=<optimized out>, keymgmt=<optimized out>, target=0x560b5b820360) at crypto/evp/keymgmt_lib.c:511
#9  EVP_PKEY_generate (ctx=ctx@entry=0x560b5b92f920, ppkey=ppkey@entry=0x7ffd42c8c0c0) at crypto/evp/pmeth_gn.c:190
#10 0x00007fb86c1d3e0e in EVP_PKEY_keygen (ctx=ctx@entry=0x560b5b92f920, ppkey=ppkey@entry=0x7ffd42c8c0c0) at crypto/evp/pmeth_gn.c:275
#11 0x00007fb86bbfaebb in generate_dh_pkey (params=0x560b5b8160b0) at /usr/src/debug/krb5-1.99.0-1.20220712gitmaster.el9.x86_64/src/plugins/preauth/pkinit/pkinit_crypto_openssl.c:723
#12 0x00007fb86bbfb055 in server_process_dh (context=context@entry=0x560b5b81b6e0, plg_cryptoctx=<optimized out>, cryptoctx=0x560b5b821120, id_cryptoctx=<optimized out>, dh_pubkey_out=dh_pubkey_out@entry=0x7ffd42c8c1d0, dh_pubkey_len_out=dh_pubkey_len_out@entry=0x7ffd42c8c198, server_key_out=0x7ffd42c8c1c8, server_key_len_out=0x7ffd42c8c19c) at /usr/src/debug/krb5-1.99.0-1.20220712gitmaster.el9.x86_64/src/plugins/preauth/pkinit/pkinit_crypto_openssl.c:3095
#13 0x00007fb86bc011d3 in pkinit_server_return_padata (context=0x560b5b81b6e0, padata=0x560b5b92b870, req_pkt=0x560b5b81d800, request=0x560b5b92e240, reply=<optimized out>, encrypting_key=<optimized out>, send_pa=0x7ffd42c8c2e0, cb=0x560b5a2f7040 <callbacks>, rock=0x560b5b81e418, moddata=0x560b5b81c6d0, modreq=0x560b5b7fc970) at /usr/src/debug/krb5-1.99.0-1.20220712gitmaster.el9.x86_64/src/plugins/preauth/pkinit/pkinit_srv.c:832
#14 0x0000560b5a2da8cf in return_padata (context=0x560b5b81b6e0, rock=rock@entry=0x560b5b81e418, req_pkt=0x560b5b81d800, request=request@entry=0x560b5b92e240, reply=reply@entry=0x560b5b81e488, encrypting_key=encrypting_key@entry=0x560b5b81e3d0, padata_context=0x560b5b81e520) at /usr/src/debug/krb5-1.99.0-1.20220712gitmaster.el9.x86_64/src/kdc/kdc_preauth.c:1457
#15 0x0000560b5a2db35e in finish_process_as_req (state=state@entry=0x560b5b81e290, errcode=errcode@entry=0) at /usr/src/debug/krb5-1.99.0-1.20220712gitmaster.el9.x86_64/src/kdc/do_as_req.c:250
#16 0x0000560b5a2dba9a in finish_preauth (arg=arg@entry=0x560b5b81e290, code=0) at /usr/src/debug/krb5-1.99.0-1.20220712gitmaster.el9.x86_64/src/kdc/do_as_req.c:471
#17 0x0000560b5a2dab41 in finish_check_padata (state=state@entry=0x560b5b82a1c0, code=0) at /usr/src/debug/krb5-1.99.0-1.20220712gitmaster.el9.x86_64/src/kdc/kdc_preauth.c:1206
#18 0x0000560b5a2dae67 in finish_verify_padata (arg=0x560b5b82a1c0, code=0, modreq=<optimized out>, e_data=0x0, authz_data=0x0) at /usr/src/debug/krb5-1.99.0-1.20220712gitmaster.el9.x86_64/src/kdc/kdc_preauth.c:1281
#19 0x00007fb86bc0031d in pkinit_server_verify_padata (context=0x560b5b81b6e0, req_pkt=<optimized out>, request=<optimized out>, enc_tkt_reply=0x560b5b81e2c8, data=<optimized out>, cb=0x560b5a2f7040 <callbacks>, rock=0x560b5b81e418, moddata=0x560b5b81c6d0, respond=0x560b5a2dacc9 <finish_verify_padata>, arg=0x560b5b82a1c0) at /usr/src/debug/krb5-1.99.0-1.20220712gitmaster.el9.x86_64/src/plugins/preauth/pkinit/pkinit_srv.c:643
#20 0x0000560b5a2dacbb in next_padata (state=state@entry=0x560b5b82a1c0) at /usr/src/debug/krb5-1.99.0-1.20220712gitmaster.el9.x86_64/src/kdc/kdc_preauth.c:1317
#21 0x0000560b5a2dafa4 in check_padata (context=0x560b5b81b6e0, rock=rock@entry=0x560b5b81e418, req_pkt=0x560b5b81d800, request=0x560b5b92e240, enc_tkt_reply=enc_tkt_reply@entry=0x560b5b81e2c8, padata_context=padata_context@entry=0x560b5b81e520, e_data=0x560b5b81e478, typed_e_data=0x560b5b81e480, respond=0x560b5a2dba57 <finish_preauth>, arg=0x560b5b81e290) at /usr/src/debug/krb5-1.99.0-1.20220712gitmaster.el9.x86_64/src/kdc/kdc_preauth.c:1376
#22 0x0000560b5a2ddf3c in process_as_req (request=<optimized out>, req_pkt=req_pkt@entry=0x560b5b81d800, local_addr=local_addr@entry=0x560b5b81d7e8, remote_addr=remote_addr@entry=0x560b5b81d6c0, kdc_active_realm=0x560b5b73f220, vctx=vctx@entry=0x560b5b7dd7b0, respond=0x560b5a2d5d5d <finish_dispatch_cache>, arg=0x560b5b824b60) at /usr/src/debug/krb5-1.99.0-1.20220712gitmaster.el9.x86_64/src/kdc/do_as_req.c:766
#23 0x0000560b5a2e07e1 in dispatch (cb=0x560b5a2f7290 <shandle>, local_addr=0x560b5b81d7e8, remote_addr=0x560b5b81d6c0, pkt=0x560b5b81d800, is_tcp=1, vctx=0x560b5b7dd7b0, respond=0x560b5a2e6c62 <process_tcp_response>, arg=0x560b5b81d750) at /usr/src/debug/krb5-1.99.0-1.20220712gitmaster.el9.x86_64/src/kdc/dispatch.c:170
#24 0x0000560b5a2e7075 in process_tcp_connection_read (ctx=0x560b5b7dd7b0, ev=<optimized out>) at ../lib/apputils/net-server.c:1362
#25 0x00007fb86c6cc7fc in verto_fire (ev=0x560b5b742700) at /usr/src/debug/libverto-0.3.2-3.el9.x86_64/src/verto.c:1020
#26 0x00007fb86bcf5d83 in ev_invoke_pending (loop=0x7fb86bd01060 <default_loop_struct>) at /usr/src/debug/libev-4.33-5.el9.x86_64/ev.c:3770
#27 0x00007fb86bcf9830 in ev_run (flags=0, loop=0x7fb86bd01060 <default_loop_struct>) at /usr/src/debug/libev-4.33-5.el9.x86_64/ev.c:4190
#28 ev_run (loop=0x7fb86bd01060 <default_loop_struct>, flags=0) at /usr/src/debug/libev-4.33-5.el9.x86_64/ev.c:4021
#29 0x0000560b5a2e9d34 in main (argc=3, argv=<optimized out>) at /usr/src/debug/krb5-1.99.0-1.20220712gitmaster.el9.x86_64/src/kdc/main.c:1039

I don't know this in invalid q size is coming from the Heimdal client or MIT KDC/OpenSSL yet.

Versions are:
openssl-3.0.1-36.el9_0
heimdal-7.7.0-11.fc36


[1] https://github.com/krb5/krb5/blob/krb5-1.19.1-final/src/plugins/preauth/pkinit/pkinit_crypto_openssl.c#L2735
[2] https://github.com/krb5/krb5/blob/krb5-1.19.1-final/src/plugins/preauth/pkinit/pkinit.h#L61

Heimdal supports 2 MODP groups by default:

* RFC2412's well-known group 2[1]
* RFC3526's well-known group 14[2] (surprisingly, the RFC mentions the size of this group is 2048, but in Heimdal the size is written to be 1760)

This issue seems related to the definition of the fits group[3]. This group is the second element of the builtin list[4], and is picked by default if the "libdefaults.pkinit_dh_min_bits" parameter is undefined equal to 0[5]. But if the parameter is set, then it will pick the first group with a size strictly larger.

Since RFC3526's well-known group 14 is the larger and first group of the list, it will be selected if pkinit_dh_min_bits is between 1 and 1759. Any other value will result in a "Did not find a DH group parameter matching requirement" error[6]. This may be a bug, because it would make more sense to sort the list in ascending size order.

To summarize, the fix is to set the libdefaults.pkinit_dh_min_bits parameter in Heimdal client configuration to any value between 1 and 1759 (ideally >1024, to be sure the group 14 will be selected rather than the group 2). It is still unclear why OpenSSL read the q prime as being 1023 bits long when Heimdal is using the group 2.


[1] https://datatracker.ietf.org/doc/html/rfc2412#appendix-E.2
[2] https://datatracker.ietf.org/doc/html/rfc3526#section-3
[3] https://github.com/heimdal/heimdal/blob/heimdal-7.7.0/lib/krb5/pkinit.c#L2092-L2112
[4] https://github.com/heimdal/heimdal/blob/heimdal-7.7.0/lib/krb5/pkinit.c#L2171
[5] https://github.com/heimdal/heimdal/blob/heimdal-7.7.0/lib/krb5/pkinit.c#L118
[6] https://github.com/heimdal/heimdal/blob/heimdal-7.7.0/lib/krb5/pkinit.c#L128-L131

PKINIT RFC[1] mentions MODP groups 4[2] and 12[3] must be supported, and group 16[4] should be supported. MIT krb5 implements these 3 groups[5], while Heimdal supports the 2 first ones only[6].

However OpenSSL only supports MODP groups from RFC3526[7], RFC5114[8], and RFC7919[9], which are not including MODP group 2. As a consequence, the Diffie-Hellman parameter is interpreted as a generated group (i.e. not one of the well-known group), and therefore goes through a verification process first.

There are 2 issues with this verification process:

* There is a bug caused by a confusion between bit and byte size[10]: the default_mdname function is supposed to pick the appropriate message digest function based on the size of the q prime in bytes, but the size passed is in bits.
* Fixing the previous bug is not enough because the byte size in this case is 128, which is not mapped to any MD function.

On Heimdal side, there are 2 points I find confusing: 

* A group is selected only if its size is strictly greater than the "pkinit_dh_min_bits" parameter[11], but the naming suggests that the configured value itself should be acceptable.
* The Heimdal definition of MODP group 14 has size 1760[12], while its definition in the RFC has size 2048.


[1] https://datatracker.ietf.org/doc/html/rfc4556#page-13
[2] https://datatracker.ietf.org/doc/html/rfc2409#section-6.4
[3] https://datatracker.ietf.org/doc/html/rfc3526#section-3
[4] https://datatracker.ietf.org/doc/html/rfc3526#section-5
[5] https://github.com/krb5/krb5/blob/krb5-1.20-final/src/plugins/preauth/pkinit/pkinit_constants.c#L294-L296
[6] https://github.com/heimdal/heimdal/blob/heimdal-7.7.0/lib/krb5/pkinit.c#L2092-L2144
[7] https://datatracker.ietf.org/doc/html/rfc3526
[8] https://datatracker.ietf.org/doc/html/rfc5114#section-2
[9] https://datatracker.ietf.org/doc/html/rfc7919#appendix-A
[10] https://github.com/openssl/openssl/blob/openssl-3.0.5/crypto/ffc/ffc_params_generate.c#L541
[11] https://github.com/heimdal/heimdal/blob/heimdal-7.7.0/lib/krb5/pkinit.c#L124
[12] https://github.com/heimdal/heimdal/blob/heimdal-7.7.0/lib/krb5/pkinit.c#L2118

(In reply to Julien Rische from comment #3)
> PKINIT RFC[1] mentions MODP groups 4[2] and 12[3] must be supported, and
> group 16[4] should be supported. MIT krb5 implements these 3 groups[5],
> while Heimdal supports the 2 first ones only[6].

My bad, I made a confusion here: mandatory groups are 2[1] and 14[2], and the optional one is 16[3].

[1] https://datatracker.ietf.org/doc/html/rfc2409#section-6.2
[2] https://datatracker.ietf.org/doc/html/rfc3526#section-3
[3] https://datatracker.ietf.org/doc/html/rfc3526#section-5

I opened 3 issues in Heimdal project:

https://github.com/heimdal/heimdal/issues/1001
https://github.com/heimdal/heimdal/issues/1002
https://github.com/heimdal/heimdal/issues/1003

I opened a feature request[1] for MODP group 2 support in OpenSSL and a bug report[2] about the bit/byte confusion in FFDH parameter verification process.

[1] https://github.com/openssl/openssl/issues/18981
[2] https://github.com/openssl/openssl/issues/18980

The KDC callstack from my previous message was using master branch from MIT krb5. This callstack demonstrating the same issue with krb5-1.19.1-16.debug.el9_0.x86_64 (current NVR on RHEL 9.0):

#0  default_mdname (N=1023) at crypto/ffc/ffc_params_generate.c:448
#1  ossl_ffc_params_FIPS186_4_gen_verify (libctx=0x0, params=0x7fff2004fb30, mode=0, type=1, L=1024, N=1023, res=0x7fff2004fb2c, cb=0x0) at crypto/ffc/ffc_params_generate.c:541
#2  0x00007f01b41f8403 in ossl_ffc_params_FIPS186_4_validate.constprop.0 (libctx=libctx@entry=0x0, params=params@entry=0x7fff2004fb30, type=type@entry=1, res=res@entry=0x7fff2004fb2c, cb=0x0) at crypto/ffc/ffc_params_validate.c:64
#3  0x00007f01b41155bc in ossl_ffc_params_simple_validate (libctx=0x0, params=0x56522df01ac8, paramstype=1, res=0x7fff2004fb2c) at crypto/ffc/ffc_params_validate.c:120
#4  0x00007f01b409602f in generate_key (dh=0x56522df01ac0) at crypto/dh/dh_key.c:337
#5  generate_key (dh=0x56522df01ac0) at crypto/dh/dh_key.c:256
#6  0x00007f01b3a58cc6 in server_process_dh (context=context@entry=0x56522de5fc30, plg_cryptoctx=<optimized out>, cryptoctx=<optimized out>, id_cryptoctx=<optimized out>, data=<optimized out>, data_len=132, dh_pubkey_out=0x7fff2004fd00, dh_pubkey_len_out=0x7fff2004fcc8, server_key_out=0x7fff2004fcf8, server_key_len_out=0x7fff2004fccc) at /usr/src/debug/krb5-1.19.1-16.debug.el9_0.x86_64/src/plugins/preauth/pkinit/pkinit_crypto_openssl.c:2894
#7  0x00007f01b3a60279 in pkinit_server_return_padata (context=0x56522de5fc30, padata=<optimized out>, req_pkt=0x56522defa340, request=0x56522df0add0, reply=<optimized out>, encrypting_key=0x56522de0ab90, send_pa=0x7fff2004fdf0, cb=0x56522dae9040 <callbacks>, rock=0x56522de0abd8, moddata=0x56522ddebf40, modreq=0x56522def7610) at /usr/src/debug/krb5-1.19.1-16.debug.el9_0.x86_64/src/plugins/preauth/pkinit/pkinit_srv.c:841
#8  0x000056522dace913 in return_padata (context=0x56522de5fc30, rock=rock@entry=0x56522de0abd8, req_pkt=0x56522defa340, request=request@entry=0x56522df0add0, reply=reply@entry=0x56522de0ac48, encrypting_key=encrypting_key@entry=0x56522de0ab90, padata_context=0x56522de0ace0) at /usr/src/debug/krb5-1.19.1-16.debug.el9_0.x86_64/src/kdc/kdc_preauth.c:1439
#9  0x000056522dacf3d6 in finish_process_as_req (state=state@entry=0x56522de0aa50, errcode=errcode@entry=0) at /usr/src/debug/krb5-1.19.1-16.debug.el9_0.x86_64/src/kdc/do_as_req.c:248
#10 0x000056522dacfaee in finish_preauth (arg=arg@entry=0x56522de0aa50, code=0) at /usr/src/debug/krb5-1.19.1-16.debug.el9_0.x86_64/src/kdc/do_as_req.c:465
#11 0x000056522daceb85 in finish_check_padata (state=state@entry=0x56522df01320, code=0) at /usr/src/debug/krb5-1.19.1-16.debug.el9_0.x86_64/src/kdc/kdc_preauth.c:1188
#12 0x000056522dacee57 in finish_verify_padata (arg=0x56522df01320, code=0, modreq=<optimized out>, e_data=0x0, authz_data=0x0) at /usr/src/debug/krb5-1.19.1-16.debug.el9_0.x86_64/src/kdc/kdc_preauth.c:1263
#13 0x00007f01b3a5bcd8 in pkinit_server_verify_padata (context=0x56522de5fc30, req_pkt=<optimized out>, request=0x56522df0add0, enc_tkt_reply=0x56522de0aa88, data=0x56522de04ab0, cb=<optimized out>, rock=0x56522de0abd8, moddata=0x56522ddebf40, respond=0x56522dacecc1 <finish_verify_padata>, arg=0x56522df01320) at /usr/src/debug/krb5-1.19.1-16.debug.el9_0.x86_64/src/plugins/preauth/pkinit/pkinit_srv.c:641
#14 0x000056522dacecb3 in next_padata (state=0x56522df01320) at /usr/src/debug/krb5-1.19.1-16.debug.el9_0.x86_64/src/kdc/kdc_preauth.c:1299
#15 0x000056522dacef69 in check_padata (context=0x56522de5fc30, rock=rock@entry=0x56522de0abd8, req_pkt=0x56522defa340, request=<optimized out>, enc_tkt_reply=enc_tkt_reply@entry=0x56522de0aa88, padata_context=padata_context@entry=0x56522de0ace0, e_data=0x56522de0ac38, typed_e_data=0x56522de0ac40, respond=0x56522dacfaab <finish_preauth>, arg=0x56522de0aa50) at /usr/src/debug/krb5-1.19.1-16.debug.el9_0.x86_64/src/kdc/kdc_preauth.c:1358
#16 0x000056522dad1b8b in process_as_req (request=<optimized out>, req_pkt=req_pkt@entry=0x56522defa340, local_addr=local_addr@entry=0x56522defa328, remote_addr=remote_addr@entry=0x56522de09ab0, kdc_active_realm=0x56522dfa8550, vctx=vctx@entry=0x56522de04ad0, respond=0x56522dac9c42 <finish_dispatch_cache>, arg=0x56522de0c700) at /usr/src/debug/krb5-1.19.1-16.debug.el9_0.x86_64/src/kdc/do_as_req.c:775
#17 0x000056522dad448c in dispatch (cb=0x56522dae9290 <shandle>, local_addr=0x56522defa328, remote_addr=0x56522de09ab0, pkt=0x56522defa340, is_tcp=1, vctx=0x56522de04ad0, respond=0x56522dad9f17 <process_tcp_response>, arg=0x56522defa290) at /usr/src/debug/krb5-1.19.1-16.debug.el9_0.x86_64/src/kdc/dispatch.c:201
#18 0x000056522dada32a in process_tcp_connection_read (ctx=0x56522de04ad0, ev=<optimized out>) at ../lib/apputils/net-server.c:1359
#19 0x00007f01b46067fc in verto_fire () from /lib64/libverto.so.1
#20 0x00007f01b3b4dd83 in ev_invoke_pending () from /lib64/libev.so.4
#21 0x00007f01b3b51830 in ev_run () from /lib64/libev.so.4
#22 0x000056522dadd069 in main (argc=3, argv=<optimized out>) at /usr/src/debug/krb5-1.19.1-16.debug.el9_0.x86_64/src/kdc/main.c:1067

These two Heimdal configuration issues where fixed upstream:
https://github.com/heimdal/heimdal/issues/1001
https://github.com/heimdal/heimdal/issues/1002

Issue migration from Bugzilla to Jira is in process at this time. This will be the last message in Jira copied from the Bugzilla bug.

This BZ has been automatically migrated to the issues.redhat.com Red Hat Issue Tracker. All future work related to this report will be managed there.

Due to differences in account names between systems, some fields were not replicated.  Be sure to add yourself to Jira issue's "Watchers" field to continue receiving updates and add others to the "Need Info From" field to continue requesting information.

To find the migrated issue, look in the "Links" section for a direct link to the new issue location. The issue key will have an icon of 2 footprints next to it, and begin with "RHEL-" followed by an integer.  You can also find this issue by visiting https://issues.redhat.com/issues/?jql= and searching the "Bugzilla Bug" field for this BZ's number, e.g. a search like:

"Bugzilla Bug" = 1234567

In the event you have trouble locating or viewing this issue, you can file an issue by sending mail to rh-issues. You can also visit https://access.redhat.com/articles/7032570 for general account information.