Bug 2179163

Summary: Openstack deploy with TLS-e when IPA-realm and IPA-domain do not match
Product: Red Hat OpenStack Reporter: Conrado Gusso Bozza <cgussobo>
Component: ansible-tripleo-ipaAssignee: Roger Heslop <rheslop>
Status: CLOSED CURRENTRELEASE QA Contact: Jeremy Agee <jagee>
Severity: high Docs Contact:
Priority: unspecified    
Version: 16.2 (Train)CC: dsedgmen, dwilde, jamsmith, jmitterm, nnavarat, rheslop
Target Milestone: ---Flags: dsedgmen: needinfo-
dsedgmen: needinfo-
Target Release: ---   
Hardware: Unspecified   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-04-21 20:59:22 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Conrado Gusso Bozza 2023-03-16 20:13:00 UTC
Description of problem:
Customer is not being able to deploy the overcloud with TLS-everywhere. They followed the steps of the documentation, but looks that tripleo use the ipa-domain as ipa-realm.

Version-Release number of selected component (if applicable):
RHEL 8.4
OSP 16.2.3

How reproducible:
Always for this customer

Steps to Reproduce:
1. Follow the steps of doc:
https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.2/html/advanced_overcloud_customization/assembly_enabling-ssl-tls-on-internal-and-public-endpoints-with-identity-management#proc_implementing-tls-e-with-ansible_enabling-ssl-tls-on-internal-and-public-endpoints-with-identity-management
2. Receive a error when trying the overcloud deploy step (Configuring TLS-e on the overcloud, step 2).

Actual results:
The overcloud deploy command stops because of FATAL errors on controller and compute on step "Wait for puppet host configuration to finish".
The error is "<13>Mar 14 05:01:44 puppet-user: Error: /Stage[main]/Tripleo::Profile::Base::Certmonger_user/Tripleo::Certmonger::Qemu[qemu-nbd-client-cert]/Certmonger_certificate[qemu-nbd-client-cert]: Could not evaluate: Execution of '/usr/bin/getcert resubmit -i qemu-nbd-client-cert -f /etc/pki/libvirt-nbd/client-cert.pem -c IPA -N CN=compute-0.internalapi.example.com -K qemu/compute-0.internalapi.example.com -D compute-0.internalapi.example.com -g 2048 -w' returned 3: Resubmitting \"qemu-nbd-client-cert\" to \"IPA\".", "

The same error of resubmit for compute happen for:
ovn_controller
ovn_metadata
libvirt-client-cert
libvirt-server-cert
libvirt-vnc-server-cert
qemu-nbd-client-cert
qemu-server-cert

The same error of resubmit for cntroller happen for:
mysql
rabbitmq
novvnc-proxy
ovn-dbs
ovn_controller
neutron-ovn
ovn-octavia
httpd-ctlplane
httpd-external
httpd-internal-api
httpd-storage
httpd-storage-mgmt
libvirt-vnc-client-cert
haproxy-ctlplane-cert
haproxy-external-cert
haproxy-internal-api-cert
haproxy-storage-cert
haproxy-storage-mgmt-cert

Expected results:
Deploy overcloud with success.

Additional info:
Customer validated the undercloud with success before try to deploy the overcloud.

[root@compute ~]# getcert list
Number of certificates and requests being tracked: 7.
Request ID 'ovn_controller':
        status: CA_UNREACHABLE
        ca-error: Server at https://idm.example.com/ipa/json failed request, will retry: 4005 (The realm for the principal does not match the realm for this IPA server).
        stuck: no
        key pair storage: type=FILE,location='/etc/pki/tls/private/ovn_controller.key'
        certificate: type=FILE,location='/etc/pki/tls/certs/ovn_controller.crt'
        CA: IPA
        issuer:
        subject:
        expires: unknown
        pre-save command:
        post-save command:
        track: yes
        auto-renew: yes
...

Comment 5 David Sedgmen 2023-03-22 22:39:37 UTC
The issue was that the docs assume you will have the same domain name as your realm name. 

By default in the templates if you do not have the parameter `CertmongerKerberosRealm` set it will use the parameter `CloudDomain` to set this parameter. 
Which in turn is used to set the puppet parameter `certmonger_krb_realm`. 


/usr/share/openstack-tripleo-heat-templates/deployment/certs/certmonger-user-baremetal-puppet.yaml
~~~
outputs:
  role_data:
    description: Role data for the certmonger-user service
    value:
      service_name: certmonger_user
      config_settings:
        map_merge:
          - certmonger_ca: {get_param: CertmongerCA}
          - if:
            - internal_tls_enabled
            - tripleo::certmonger::ca::crl::crl_source: {get_param: DefaultCRLURL}
              certmonger_ca_vnc: {get_param: CertmongerVncCA}
              certmonger_ca_qemu: {get_param: CertmongerQemuCA}
              certmonger_krb_realm:
                if:
                - {not: {equals: [{get_param: CertmongerKerberosRealm},'']}}
                - {get_param: CertmongerKerberosRealm}
                - yaql:
                    expression: $.data.toUpper()
                    data: {get_param: CloudDomain}
            - {}
~~~


Finally this is used to set the realm part of the `principle` parameter of the `*certificates_specs`. 

/usr/share/openstack-tripleo-heat-templates/deployment/apache/apache-baremetal-puppet.j2.yaml
~~~
              apache_certificates_specs:
                map_merge:
                  repeat:
                    template:
                      httpd-NETWORK:
                        service_certificate: '/etc/pki/tls/certs/httpd/httpd-NETWORK.crt'
                        service_key: '/etc/pki/tls/private/httpd/httpd-NETWORK.key'
                        hostname: "%{hiera('fqdn_NETWORK')}"
                        principal: "HTTP/%{hiera('fqdn_NETWORK')}@%{hiera('certmonger_krb_realm')}"
                        postsave_cmd: "pkill -USR1 httpd"
                        key_size:
                          if:
                            - key_size_override_unset
                            - {get_param: CertificateKeySize}
                            - {get_param: ApacheCertificateKeySize}
                    for_each:
                      NETWORK: {get_attr: [ApacheNetworks, value]}
            - {}
~~~


Which is what is used for creating the certmonger certificate requests.

/usr/share/openstack-puppet/modules/tripleo/manifests/profile/base/certmonger_user.pp
~~~
    $apache_certificates_specs_filtered = $apache_certificates_specs.filter | $specs, $keys | { ! empty($keys[hostname]) }
    unless empty($apache_certificates_specs_filtered) {
      include ::tripleo::certmonger::apache_dirs
      ensure_resources('tripleo::certmonger::httpd', $apache_certificates_specs_filtered)
    }
~~~
/usr/share/openstack-puppet/modules/tripleo/manifests/certmonger/httpd.pp
~~~
define tripleo::certmonger::httpd (
  $hostname,
  $service_certificate,
  $service_key,
  $certmonger_ca = hiera('certmonger_ca', 'local'),
  $dnsnames      = undef,
  $postsave_cmd  = undef,
  $principal     = undef,
  $key_size      = 2048,
) {
  include ::certmonger
  include ::apache::params

  if $dnsnames {
    $dnsnames_real = $dnsnames
  } else {
    $dnsnames_real = $hostname
  }

  certmonger_certificate { $name :
    ensure       => 'present',
    certfile     => $service_certificate,
    keyfile      => $service_key,
    hostname     => $hostname,
    dnsname      => $dnsnames_real,
    principal    => $principal,
    postsave_cmd => $postsave_cmd,
    ca           => $certmonger_ca,
    key_size     => $key_size,
    wait         => true,
    tag          => 'apache-cert',
    require      => Class['::certmonger'],
    subscribe    => File[$service_key],
  }
  file { $service_key:
    audit => [content],
  }

  Certmonger_certificate[$name] ~> Service<| title == $::apache::params::service_name |>
}
~~~