Python version:

# rpm -q platform-python
platform-python-3.6.8-11.el8.x86_64

I'd say this is a candidate for blocker considering it blocks registration, a core piece of functionality necessary to receive updates.

Moving to python3 based on the investigation done in comment 7 for a fix.

We're investigating OpenSSL failures under FIPS, but aren't very familiar with OpenSSL. The "malloc failure" error is not very enlightening. Help would be welcome.

Hello, Tomáš, TL;DR at the end.

The malloc issue is reproducible also on Fedora 30.
It seems to be solved in rawhide, where there is openssl-1.1.1c-9.fc31.x86_64.

The line mentioned in comment #7:
 839         ret = SSL_do_handshake(self->ssl);
calls OpenSSL mechanics to do the handshake.

Given this fact and given that updated OpenSSL solves the malloc issue, this one is most likely an OpenSSL issue.

However, there is one more issue:

[root@0c43a8c58ae4 cpython]# ./python -m test test_ssl -m test_check_hostname
Run tests sequentially
0:00:00 load avg: 0.70 [1/1] test_ssl
test test_ssl failed -- Traceback (most recent call last):
  File "/root/cpython/Lib/test/test_ssl.py", line 2545, in test_check_hostname
    s.connect((HOST, server.port))
  File "/root/cpython/Lib/ssl.py", line 1109, in connect
    self._real_connect(addr, False)
  File "/root/cpython/Lib/ssl.py", line 1100, in _real_connect
    self.do_handshake()
  File "/root/cpython/Lib/ssl.py", line 1077, in do_handshake
    self._sslobj.do_handshake()
  File "/root/cpython/Lib/ssl.py", line 689, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: TLSV1_ALERT_INTERNAL_ERROR] tlsv1 alert internal error (_ssl.c:852)

test_ssl failed

I am still investigating whether this is also an OpenSSL's fault or Python's one.
Debugging has brought me very deep thtough OpenSSL internals, so chances are that this fault is also on OpenSSL's side.
It uses MD5_SHA1 message digest even though MD5 shouldn't be allowed at all and I fail to find any instruction to use MD5 from Python's side (a well hidden default in either Python or OpenSSL code).

This place seems to be where all the magic happens.
openssl-1.1.1c/ssl/s3_enc.c:

376 int ssl3_digest_cached_records(SSL *s, int keep)
377 {
378     const EVP_MD *md;
379     long hdatalen;
380     void *hdata;  
381 
382     if (s->s3->handshake_dgst == NULL) {
383         hdatalen = BIO_get_mem_data(s->s3->handshake_buffer, &hdata);
384         if (hdatalen <= 0) {
385             SSLfatal(s, SSL_AD_INTERNAL_ERROR, SSL_F_SSL3_DIGEST_CACHED_RECORDS,
386                      SSL_R_BAD_HANDSHAKE_LENGTH);
387             return 0;
388         }
389 
390         s->s3->handshake_dgst = EVP_MD_CTX_new();
391         if (s->s3->handshake_dgst == NULL) {
392             SSLfatal(s, SSL_AD_INTERNAL_ERROR, SSL_F_SSL3_DIGEST_CACHED_RECORDS,
393                      ERR_R_MALLOC_FAILURE);  // <- the malloc failure from earlier
394             return 0;
395         }
396 
397         md = ssl_handshake_md(s);
398         if (md == NULL || !EVP_DigestInit_ex(s->s3->handshake_dgst, md, NULL)
399             || !EVP_DigestUpdate(s->s3->handshake_dgst, hdata, hdatalen)) {
400             SSLfatal(s, SSL_AD_INTERNAL_ERROR, SSL_F_SSL3_DIGEST_CACHED_RECORDS,
401                      ERR_R_INTERNAL_ERROR); // <- the internal error I face now
402             return 0;
403         }
404     }
405     if (keep == 0) {
406         BIO_free(s->s3->handshake_buffer);
407         s->s3->handshake_buffer = NULL;
408     }
409 
410     return 1;
411 }

Note the comments on lines 393 and 401.

I'll continue to investigate and report here afterwards.

TL;DR:

On the line:
397         md = ssl_handshake_md(s);

in openssl-1.1.1c/ssl/s3_enc.c, the message digest gets returned as follows:

(gdb) p md
$3 = (const EVP_MD *) 0x7f0641a64740 <md5_sha1_md>

This gets detected and blocked by FIPS patch code inside openssl-1.1.1c/crypto/evp/digest.c:
132         if (FIPS_mode()) {
133             if (!(type->flags & EVP_MD_FLAG_FIPS)
134                 && !(ctx->flags & EVP_MD_CTX_FLAG_NON_FIPS_ALLOW)) {
135                 EVPerr(EVP_F_EVP_DIGESTINIT_EX, EVP_R_DISABLED_FOR_FIPS);
136                 return 0;
137             }
138         }

Tomáš, do you think this could be an OpenSSL's fault?

Can you please retest with the latest RHEL-8.1.0 compose which has openssl-1.1.1c-2.el8 build?

The test case from comment #10 apparently uses TLS-1.0 which is not FIPS compliant anymore. So that failure is irrelevant in the FIPS mode and should not be related to the original failure.

Retesting the original failing scenario from comment 0 and comment 3 against the RHEL-8.1.0-Snapshot-2.1 compose RHEL-8.1.0-20190821.0...

[root@kvm-01-guest07 ~]# rpm -q subscription-manager openssl crypto-policies kernel
subscription-manager-1.25.14-1.el8.x86_64
openssl-1.1.1c-2.el8.x86_64                <============ matches comment 11
crypto-policies-20190807-1.git9b1477b.el8.noarch
kernel-4.18.0-135.el8.x86_64
[root@kvm-01-guest07 ~]# 
[root@kvm-01-guest07 ~]# update-crypto-policies --show
FIPS
[root@kvm-01-guest07 ~]# sysctl crypto.fips_enabled
crypto.fips_enabled = 1
[root@kvm-01-guest07 ~]# getenforce
Enforcing
[root@kvm-01-guest07 ~]# 
[root@kvm-01-guest07 ~]# subscription-manager register --auto-attach
Registering to: subscription.rhsm.redhat.com:443/subscription
Username: rhelentqe
Password: 
The system has been registered with ID: f2363afb-7b8e-4d9e-900d-ffb42f71d1d2
The registered system name is: kvm-01-guest07.lab.eng.rdu2.redhat.com
Installed Product Current Status:
Product Name: Red Hat Enterprise Linux for x86_64 High Touch Beta
Status:       Subscribed



VERIFIED: The registration is successful and is no longer blocked by "Unable to verify server's identity: [SSL] malloc failure (_ssl.c:877)"

As this was only a temporary regression in the openssl-1.1.1c-1.el8 build I am going to close it as CURRENTRELEASE.