I tried reproducing your issue and even though I managed to do that, I am getting a different output.

When asking certtool about the subject, I get this:

  CN=rhel9,O=Libvirt Project,L=London,ST=London,C=GB

And when libvirt complains about the DN being wrong, I get this:

  C=GB,ST=London,L=London,O=Libvirt Project,CN=rhel9

Which is switched around compared to your report.

I have not dug much more deep, but gnutls_x509_crt_get_dn(3), which we are using, is saying:

  The name will be in the form "C=xxxx,O=yyyy,CN=zzzz" as described in RFC4514.

And we will have to rely on that, unless we want to do some extra matching, which could be potentially dangerous.

My guess is that gnutls maybe have switched the output at some point.  What version of gnutls are you using?  Mine are:

  # rpm -qa 'gnutls*'
  gnutls-3.7.2-4.el9.x86_64
  gnutls-dane-3.7.2-4.el9.x86_64
  gnutls-utils-3.7.2-4.el9.x86_64

I have a documentation fix for this issue (explaining a bit more in the configuration file), but I guess this might be worth asking them, or even creating a bug there (unless you have a version with a bug which is fixed in mine).  Anyway I do not see why certtool would output that information with the fields in different order anyway.

Few more things:

I can confirm certtool changed how it prints the Subject.  In RHEL8 (gnutls*-3.6.14-7.el8_3) it used to print this:

  C=GB,ST=London,L=London,O=Libvirt Project,CN=rhel9

which works with what libvirt expects because it is what gnutls_x509_crt_get_dn(3) returns.

Unfortunately RFC4514 does explicitly say that the order of the fields can be in any order.  So it is not an error in certtool, just an issue with libvirt depending on the consistency of its output.

This cannot be fixed correctly with the current way of configuring this and in my humble opinion this configuration knob did not ever make sense.  I will raise the question upstream and update this bug.

From my testing it is more complicated, because it depends both on which version of certtool created the cert and which version is querying the cert.

I created 3 certs on rhel7, rhel8 and rhel9 using certool, then queried them using gnutls_x509_crt_get_dn

  created-rhel7:  CN=acme-client1,O=ACME,L=London,ST=London,C=GB
  created-rhel8:  CN=acme-client1,O=ACME,L=London,ST=London,C=GB
  created-rhel9:  C=GB,ST=London,L=London,O=ACME,CN=acme-client1

I also queried using $ openssl x509 -in usercert.pem -noout -subject | sed 's/subject= //'

  created-rhel7:  subject=CN = acme-client1, O = ACME, L = London, ST = London, C = GB
  created-rhel8:  subject=CN = acme-client1, O = ACME, L = London, ST = London, C = GB
  created-rhel9:  subject=C = GB, ST = London, L = London, O = ACME, CN = acme-client1


I then also queried them using 'certtool -i --infile PEMFILE | grep Subject:', on each RHEL version

gnutls certtool rhel7

  created-rhel7:  Subject: CN=acme-client1,O=ACME,L=London,ST=London,C=GB
  created-rhel8:  Subject: CN=acme-client1,O=ACME,L=London,ST=London,C=GB
  created-rhel9:  Subject: C=GB,ST=London,L=London,O=ACME,CN=acme-client1

gnutls certtool rhel8

  created-rhel7:  Subject: C=GB,ST=London,L=London,O=ACME,CN=acme-client1
  created-rhel8:  Subject: C=GB,ST=London,L=London,O=ACME,CN=acme-client1
  created-rhel9:  Subject: CN=acme-client1,O=ACME,L=London,ST=London,C=GB

gnutls certtool rhel9

  created-rhel7:  Subject: C=GB,ST=London,L=London,O=ACME,CN=acme-client1
  created-rhel8:  Subject: C=GB,ST=London,L=London,O=ACME,CN=acme-client1
  created-rhel9:  Subject: CN=acme-client1,O=ACME,L=London,ST=London,C=GB


What we can see from this:

 - In RHEL-8 gnutls certtool reversed the order in which it reports distinguished name fields (see https://bugzilla.redhat.com/show_bug.cgi?id=1394318)
 - In RHEL-9 gnutls certtool reversed the order in which it encoded distinguished name fields in the certificate


The gnutls_x509_crt_get_dn() API libvirt uses always matches the order of the fields in the certificate

This is good because it means from a functional POV, libvirt behaviour has not changed in RHEL-9, we are still honouring the order in the certificate when doing wildcard matches.


In terms of certificates people use in production deployments, there has never been any guarantee about what order fields will appear in. That is entirely dependant on what tools people use to create certificates for use with libvirt. They might be using certtool, or openssl, or FreeIPA, or something else again.

What matters is that the "tls_allowed_dn_list" setting applies to the actual order in the certificate and that is still working correctly and has not changed in behaviour.

The main bugs here are

 - Our documentation tells people to look at the 'certtool' output to determine the string they need to use for the match.

   This is wrong. We need to tell people to use the *reverse* of what certtool reports. 

 - QE was creating a certificate on RHEL-9 with certtool, and then using the tls_allowed_dn_list setting their test plan had used in RHEL-8. 

   This doesn't take into account the fact that certtool has created the cert in a different way. The test plan needs updating for this. A certificate created on RHEL-8 and used on RHEL-9 would still have worked, but a cert created on RHEL-9 needs a reversed setting

Fixed upstream by not recommending certtool, but rather our own helper virt-pki-query-dn.

Fixed upstream with v7.9.0-124-g971504354405, v7.9.0-125-g5d972ad910c1, v7.9.0-126-g831f5415826b:
commit 971504354405de286a3bb1967763ae3fc02b8dc6
Author: Martin Kletzander <mkletzan>
Date:   Thu Nov 11 15:35:38 2021 +0100

    tools: Add virt-pki-query-dn binary
    
commit 5d972ad910c1ffdf11fec7482db805329e8a01fd
Author: Martin Kletzander <mkletzan>
Date:   Thu Nov 11 15:56:05 2021 +0100

    Add suggestions for virt-pki-query-dn usage
    
commit 831f5415826bb0792c28d50544e6970df4443d66
Author: Martin Kletzander <mkletzan>
Date:   Thu Nov 11 15:59:48 2021 +0100

    news: Mention the addition of virt-pki-query-dn binary

Test with libvirt-7.10.0-1.el9.x86_64:

Steps to Reproduce:
1. setup the tls env

2. connect to the libvirtd daemon via tls transport

# virsh -c qemu+tls://$hostname/system
Welcome to virsh, the virtualization interactive terminal.

Type:  'help' for help with commands
       'quit' to quit

virsh # exit


3. check the subject from clientcert.pem
# virt-pki-query-dn clientcert.pem 
CN=$client_hostname,O=Red Hat,L=London,ST=London,C=GB

4. config the config file
# cat /etc/libvirt/libvirtd.conf
tls_allowed_dn_list = ["CN=$client_hostname,O=Red Hat,L=London,ST=London,C=GB"]

5. connect to the libvirtd daemon via tls transport
# virsh -c qemu+tls://$hostname/system
Welcome to virsh, the virtualization interactive terminal.

Type:  'help' for help with commands
       'quit' to quit

virsh #

Verify this bug with:
libvirt-7.10.0-1.el9.x86_64

1. setup the tls env

2. connect to the libvirtd daemon via tls transport

# virsh -c qemu+tls://$hostname/system
Welcome to virsh, the virtualization interactive terminal.

Type:  'help' for help with commands
       'quit' to quit

virsh # exit


3. check the subject from clientcert.pem
# virt-pki-query-dn clientcert.pem
C=GB,ST=London,L=London,O=Red Hat,CN=$client_hostname

4. config the config file
# cat /etc/libvirt/libvirtd.conf
tls_allowed_dn_list = ["C=GB,ST=London,L=London,O=Red Hat,CN=$client_hostname"]

5. connect to the libvirtd daemon via tls transport
# virsh -c qemu+tls://$server_hostname/system
Welcome to virsh, the virtualization interactive terminal.

Type:  'help' for help with commands
       'quit' to quit

virsh #

6. change the config file
# cat /etc/libvirt/libvirtd.conf
tls_allowed_dn_list = ["C=GB,ST=London,L=London,O=Red Hat,CN=$wrong_client_hostname"]

7. # virsh -c qemu+tls://$server_hostname/system
error: failed to connect to the hypervisor
error: Unable to read TLS confirmation: Input/output error

8. check the log
2021-12-26 09:39:26.803+0000: 2773553: debug : virNetTLSContextValidCertificate:1064 : Peer DN is C=GB,ST=London,L=London,O=Red Hat,CN=$client_hostname
2021-12-26 09:39:26.803+0000: 2773553: debug : virNetTLSContextCheckCertDNACL:368 : Failed ACL check for client DN 'C=GB,ST=London,L=London,O=Red Hat,CN=$client_hostname'
2021-12-26 09:39:26.803+0000: 2773553: error : virNetTLSContextCheckCertDNACL:371 : Client's Distinguished Name is not on the list of allowed clients (tls_allowed_dn_list).  Use 'virt-pki-query-dn clientcert.pem' to view the Distinguished Name field in the client certificate, or run this daemon with --verbose option.
2021-12-26 09:39:26.803+0000: 2773553: info : virNetTLSContextValidCertificate:1104 : RPC_TLS_CONTEXT_SESSION_DENY: ctxt=0x5563e42a2830 sess=0x7f6438010db0 dname=C=GB,ST=London,L=London,O=Red Hat,CN=$client_hostname
2021-12-26 09:39:26.803+0000: 2773553: warning : virNetTLSContextCheckCertificate:1126 : Certificate check failed Client's Distinguished Name is not on the list of allowed clients (tls_allowed_dn_list).  Use 'virt-pki-query-dn clientcert.pem' to view the Distinguished Name field in the client certificate, or run this daemon with --verbose option.

The method of how to get the correct DN is also reported.

As the test result matches with the expected one, mark the bug as verified.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (new packages: libvirt), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:2390