Bug 2081697

Description of problem:
when deploying a DCN site container-puppet-haproxy fails to start on  DistributedComputeHCIScaleOut deployed with TLS-E.
There are some CRL certs which are not available and puppet generated haproxy config refers to them.

The exact log:
2022-05-03 01:09:02.340647 |                                      |    WARNING | ERROR: Can't run container container-puppet-haproxy
stderr: + /usr/bin/puppet apply --summarize --detailed-exitcodes --color=false --modulepath=/etc/puppet/modules:/usr/share/openstack-puppet/modules --tags '"file,file_line,concat,augeas,cron,haproxy_config"' /etc/config.pp
+ logger -s -t puppet-user
<13>May  3 01:08:46 puppet-user: Error: Facter: error while resolving custom fact "haproxy_version": undefined method `strip' for nil:NilClass
<13>May  3 01:08:52 puppet-user: Warning: /etc/puppet/hiera.yaml: Use of 'hiera.yaml' version 3 is deprecated. It should be converted to version 5
<13>May  3 01:08:52 puppet-user:    (file: /etc/puppet/hiera.yaml)
<13>May  3 01:08:52 puppet-user: Warning: Undefined variable '::deploy_config_name';
<13>May  3 01:08:52 puppet-user:    (file & line not available)
<13>May  3 01:08:52 puppet-user: Warning: The function 'hiera' is deprecated in favor of using 'lookup'. See https://puppet.com/docs/puppet/7.10/deprecated_language.html
<13>May  3 01:08:52 puppet-user:    (file & line not available)
<13>May  3 01:08:53 puppet-user: Warning: Scope(Haproxy::Config[haproxy]): haproxy: The $merge_options parameter will default to true in the next major release. Please review the documentation regarding the implications.
<13>May  3 01:08:53 puppet-user: Notice: Compiled catalog for dcn1-computehciscaleout1-0.redhat.local in environment production in 0.88 seconds
<13>May  3 01:08:53 puppet-user: Error: Execution of '/usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg20220503-12-1npa7ww -c' returned 1: [NOTICE]   (79) : haproxy version is 2.4.7-b5e51a5
<13>May  3 01:08:53 puppet-user: [NOTICE]   (79) : path to executable is /usr/sbin/haproxy
<13>May  3 01:08:53 puppet-user: [ALERT]    (79) : parsing [/etc/haproxy/haproxy.cfg20220503-12-1npa7ww:48] : 'server dcn1-computehci1-0.internalapi.redhat.local' : 'crl-file' : unable to load /etc/pki/CA/crl/overcloud-crl.pem.
<13>May  3 01:08:53 puppet-user: [ALERT]    (79) : parsing [/etc/haproxy/haproxy.cfg20220503-12-1npa7ww:49] : 'server dcn1-computehci1-1.internalapi.redhat.local' : 'crl-file' : unable to load /etc/pki/CA/crl/overcloud-crl.pem.
<13>May  3 01:08:53 puppet-user: [ALERT]    (79) : parsing [/etc/haproxy/haproxy.cfg20220503-12-1npa7ww:50] : 'server dcn1-computehci1-2.internalapi.redhat.local' : 'crl-file' : unable to load /etc/pki/CA/crl/overcloud-crl.pem.
<13>May  3 01:08:53 puppet-user: [ALERT]    (79) : Error(s) found in configuration file : /etc/haproxy/haproxy.cfg20220503-12-1npa7ww
<13>May  3 01:08:53 puppet-user: [ALERT]    (79) : Fatal errors found in configuration.
<13>May  3 01:08:53 puppet-user: Error: /Stage[main]/Haproxy/Haproxy::Instance[haproxy]/Haproxy::Config[haproxy]/Concat[/etc/haproxy/haproxy.cfg]/File[/etc/haproxy/haproxy.cfg]/content: change from '{sha256}8afc9a0bcc462f08af54b6ac1cbfc3b8343b1feee00b4ab07d7a8c7b47065f0b' to '{sha256}f8be4e72c4471d14a08e5643725035032211626dc9cd5e7b54767c826cfd64ef' failed: Execution of '/usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg20220503-12-1npa7ww -c' returned 1: [NOTICE]   (79) : haproxy version is 2.4.7-b5e51a5
<13>May  3 01:08:53 puppet-user: [NOTICE]   (79) : path to executable is /usr/sbin/haproxy
<13>May  3 01:08:53 puppet-user: [ALERT]    (79) : parsing [/etc/haproxy/haproxy.cfg20220503-12-1npa7ww:48] : 'server dcn1-computehci1-0.internalapi.redhat.local' : 'crl-file' : unable to load /etc/pki/CA/crl/overcloud-crl.pem.
<13>May  3 01:08:53 puppet-user: [ALERT]    (79) : parsing [/etc/haproxy/haproxy.cfg20220503-12-1npa7ww:49] : 'server dcn1-computehci1-1.internalapi.redhat.local' : 'crl-file' : unable to load /etc/pki/CA/crl/overcloud-crl.pem.
<13>May  3 01:08:53 puppet-user: [ALERT]    (79) : parsing [/etc/haproxy/haproxy.cfg20220503-12-1npa7ww:50] : 'server dcn1-computehci1-2.internalapi.redhat.local' : 'crl-file' : unable to load /etc/pki/CA/crl/overcloud-crl.pem.
<13>May  3 01:08:53 puppet-user: [ALERT]    (79) : Error(s) found in configuration file : /etc/haproxy/haproxy.cfg20220503-12-1npa7ww
<13>May  3 01:08:53 puppet-user: [ALERT]    (79) : Fatal errors found in configuration.
<13>May  3 01:08:53 puppet-user: Notice: /Stage[main]/Haproxy/Haproxy::Instance[haproxy]/Haproxy::Config[haproxy]/Concat[/etc/haproxy/haproxy.cfg]/File[/etc/haproxy/haproxy.cfg]/mode: mode changed '0644' to '0640'
<13>May  3 01:08:53 puppet-user: Notice: Applied catalog in 0.19 seconds
<13>May  3 01:08:53 puppet-user: Application:
<13>May  3 01:08:53 puppet-user:    Initial environment: production
<13>May  3 01:08:53 puppet-user:    Converged environment: production
<13>May  3 01:08:53 puppet-user:          Run mode: user
<13>May  3 01:08:53 puppet-user: Changes:
<13>May  3 01:08:53 puppet-user:             Total: 1
<13>May  3 01:08:53 puppet-user: Events:
<13>May  3 01:08:53 puppet-user:           Failure: 1
<13>May  3 01:08:53 puppet-user:           Success: 1
<13>May  3 01:08:53 puppet-user:             Total: 2
<13>May  3 01:08:53 puppet-user: Resources:
<13>May  3 01:08:53 puppet-user:           Changed: 1
<13>May  3 01:08:53 puppet-user:            Failed: 1
<13>May  3 01:08:53 puppet-user:       Out of sync: 1
<13>May  3 01:08:53 puppet-user:           Skipped: 11
<13>May  3 01:08:53 puppet-user:             Total: 21
<13>May  3 01:08:53 puppet-user: Time:
<13>May  3 01:08:53 puppet-user:       Concat file: 0.00
<13>May  3 01:08:53 puppet-user:    Concat fragment: 0.00
<13>May  3 01:08:53 puppet-user:              File: 0.08
<13>May  3 01:08:53 puppet-user:    Transaction evaluation: 0.18
<13>May  3 01:08:53 puppet-user:    Catalog application: 0.19
<13>May  3 01:08:53 puppet-user:    Config retrieval: 0.98
<13>May  3 01:08:53 puppet-user:          Last run: 1651540133
<13>May  3 01:08:53 puppet-user:             Total: 0.19
<13>May  3 01:08:53 puppet-user: Version:
<13>May  3 01:08:53 puppet-user:            Config: 1651540132
<13>May  3 01:08:53 puppet-user:            Puppet: 7.10.0
+ rc=6
+ '[' false = false ']'
+ set +x
2022-05-03 01:09:02.344588 | 52540078-cf9a-b171-f034-000000006653 |      FATAL | Create containers managed by Podman for /var/lib/tripleo-config/container-puppet-config/step_1 | dcn1-computehciscaleout1-0 | error={"changed": false, "msg": "Failed containers: container-puppet-haproxy"}

The workaround is to set ExtraConfig for the nodes:
tripleo::haproxy::crl_file: null 

Version-Release number of selected component (if applicable):
ansible-tripleo-ipsec-11.0.1-0.20210910011424.b5559c8.el9ost.noarch
ansible-role-tripleo-modify-image-1.3.1-0.20220216001439.30d23d5.el9ost.noarch
ansible-tripleo-ipa-0.2.3-0.20220301190449.6b0ed82.el9ost.noarch
puppet-tripleo-14.2.3-0.20220407012437.87240e8.el9ost.noarch
python3-tripleo-common-15.4.1-0.20220328184445.0c754c6.el9ost.noarch
tripleo-ansible-3.3.1-0.20220407091528.0bc2994.el9ost.noarch
openstack-tripleo-validations-14.2.2-0.20220408101530.6614654.el9ost.noarch
openstack-tripleo-common-containers-15.4.1-0.20220328184445.0c754c6.el9ost.noarch
openstack-tripleo-common-15.4.1-0.20220328184445.0c754c6.el9ost.noarch
openstack-tripleo-heat-templates-14.3.1-0.20220404155604.75fd885.el9ost.noarch
python3-tripleoclient-16.4.1-0.20220407001042.0021766.el9ost.noarch

How reproducible:
Always

Steps to Reproduce:
1. Deploy DCN toology with TLS-E and ceph storage at the DCN site on DistributedComputeHCIScaleOut nodes.

Actual results:
container-puppet-haproxy fails to start and Edge haproxy is not configured

Expected results:
Successful configuration of haproxy at Edge on the DistributedComputeHCIScaleOut nodes.

Additional info: