Bug 2014107

Summary:	DCN site with TLS-E fails with error "invalid 'csr': IP address in subjectAltName (172.20.1.93) unreachable from DNS names"
Product:	Red Hat OpenStack	Reporter:	Sadique Puthen <sputhenp>
Component:	ansible-tripleo-ipa	Assignee:	Roger Heslop <rheslop>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Jeremy Agee <jagee>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	16.2 (Train)	CC:	owalsh, rheslop
Target Milestone:	---	Keywords:	Documentation, Triaged
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2021-12-01 14:34:11 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Sadique Puthen 2021-10-14 13:40:46 UTC

Description of problem:

Deploying a DCN site with TLS-E fails with error "invalid 'csr': IP address in subjectAltName (172.20.1.93) unreachable from DNS names".

puppet-user: Error: /Stage[main]/Tripleo::Certmonger::Etcd/Certmonger_certificate[etcd]: Could not evaluate: Could not get certificate: Server at https://ipa-server.redhat.local/ipa/json denied our request, giving up: 3009 (invalid 'csr': IP address in subjectAltName (172.20.1.93) unreachable from DNS names).\n<13>Oct 14 13:27:00 puppet-user: Notice: /Stage[main]/Tripleo::Certmonger::Etcd/File[/etc/pki/tls/certs/etcd.crt]:

There is no problem to deploy a central site with TLS-E.



Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 2 Sadique Puthen 2021-10-14 13:48:52 UTC

2021-10-14 09:27:39.290297 | 52540053-999d-6ecf-ddc3-00000000350a |      FATAL | Wait for puppet host configuration to finish | hci-3-edge-1 | error={"ansible_job_id": "801523124085.33243", "attempts": 12, "changed": true, "cmd": "set -o pipefail; puppet apply  --modulepath=/etc/puppet/modules:/opt/stack/puppet-modules:/usr/share/openstack-puppet/modules --detailed-exitcodes --summarize --color=false   /var/lib/tripleo-config/puppet_step_config.pp 2>&1 | logger -s -t puppet-user", "delta": "0:00:46.400588", "end": "2021-10-14 13:27:38.560974", "failed_when_result": true, "finished": 1, "msg": "non-zero return code", "rc": 6, "start": "2021-10-14 13:26:52.160386", "stderr": "<13>Oct 14 13:26:52 puppet-user: Warning: The function 'hiera' is deprecated in favor of using 'lookup'. See https://puppet.com/docs/puppet/5.5/deprecated_language.html\\n   (file & line not available)\n<13>Oct 14 13:26:57 puppet-user: Warning: /etc/puppet/hiera.yaml: Use of 'hiera.yaml' version 3 is deprecated. It should be converted to version 5\n<13>Oct 14 13:26:57 puppet-user:    (file: /etc/puppet/hiera.yaml)\n<13>Oct 14 13:26:57 puppet-user: Warning: Undefined variable '::deploy_config_name'; \\n   (file & line not available)\n<13>Oct 14 13:26:57 puppet-user: Warning: ModuleLoader: module 'tripleo' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules\\n   (file & line not available)\n<13>Oct 14 13:26:58 puppet-user: Warning: Unknown variable: '::deployment_type'. (file: /etc/puppet/modules/tripleo/manifests/profile/base/database/mysql/client.pp, line: 89, column: 8)\n<13>Oct 14 13:26:58 puppet-user: Warning: ModuleLoader: module 'concat' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules\\n   (file & line not available)\n<13>Oct 14 13:26:58 puppet-user: Warning: tag is a metaparam; this value will inherit to all contained resources in the tripleo::firewall::rule definition\n<13>Oct 14 13:26:58 puppet-user: Notice: Scope(Class[Tripleo::Firewall::Post]): At this stage, all network traffic is blocked.\n<13>Oct 14 13:26:58 puppet-user: Warning: ModuleLoader: module 'nova' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules\\n   (file & line not available)\n<13>Oct 14 13:26:58 puppet-user: Warning: ModuleLoader: module 'openstacklib' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules\\n   (file & line not available)\n<13>Oct 14 13:26:58 puppet-user: Notice: Compiled catalog for hci-3-edge-1.redhat.local in environment production in 0.96 seconds\n<13>Oct 14 13:26:59 puppet-user: Notice: /Stage[main]/Main/Package_manifest[/var/lib/tripleo/installed-packages/overcloud_HCIEdge11]/ensure: created\n<13>Oct 14 13:26:59 puppet-user: Notice: /Stage[main]/Tripleo::Certmonger::Apache_dirs/File[/etc/pki/tls/certs/httpd]/ensure: created\n<13>Oct 14 13:26:59 puppet-user: Notice: /Stage[main]/Tripleo::Certmonger::Apache_dirs/File[/etc/pki/tls/private/httpd]/ensure: created\n<13>Oct 14 13:26:59 puppet-user: Notice: /Stage[main]/Tripleo::Certmonger::Libvirt_dirs/File[/etc/pki/libvirt]/ensure: created\n<13>Oct 14 13:26:59 puppet-user: Notice: /Stage[main]/Tripleo::Certmonger::Libvirt_dirs/File[/etc/pki/libvirt/private]/ensure: created\n<13>Oct 14 13:26:59 puppet-user: Notice: /Stage[main]/Tripleo::Certmonger::Libvirt_vnc_dirs/File[/etc/pki/libvirt-vnc]/ensure: created\n<13>Oct 14 13:26:59 puppet-user: Notice: /Stage[main]/Tripleo::Certmonger::Ca::Libvirt_vnc/File[/etc/pki/libvirt-vnc/ca-cert.pem]/ensure: created\n<13>Oct 14 13:26:59 puppet-user: Notice: /Stage[main]/Tripleo::Certmonger::Qemu_dirs/File[/etc/pki/qemu]/ensure: created\n<13>Oct 14 13:26:59 puppet-user: Notice: /Stage[main]/Tripleo::Certmonger::Ca::Qemu/File[/etc/pki/qemu/ca-cert.pem]/ensure: created\n<13>Oct 14 13:26:59 puppet-user: Notice: /Stage[main]/Tripleo::Certmonger::Qemu_nbd_dirs/File[/etc/pki/libvirt-nbd]/ensure: created\n<13>Oct 14 13:26:59 puppet-user: Notice: /Stage[main]/Tripleo::Certmonger::Etcd/File[/usr/bin/certmonger-etcd-refresh.sh]/ensure: defined content as '{md5}0aa1a0196e0573519b00c15035ea4947'\n<13>Oct 14 13:26:59 puppet-user: Notice: /Stage[main]/Certmonger/Service[certmonger]/enable: enable changed 'false' to 'true'\n<13>Oct 14 13:26:59 puppet-user: Notice: /Stage[main]/Certmonger/Service[certmonger]: Triggered 'refresh' from 1 event\n<13>Oct 14 13:26:59 puppet-user: Notice: /Stage[main]/Tripleo::Certmonger::Etcd/Certmonger_certificate[etcd]/ensure: created\n<13>Oct 14 13:26:59 puppet-user: Warning: Could not get certificate: Execution of '/usr/bin/getcert request -I etcd -f /etc/pki/tls/certs/etcd.crt -c IPA -N CN=hci-3-edge-1.internalapi.redhat.local -K etcd/hci-3-edge-1.internalapi.redhat.local -D hci-3-edge-1.internalapi.redhat.local -A 172.20.1.93 -C /usr/bin/certmonger-etcd-refresh.sh -g 2048 -w -k /etc/pki/tls/private/etcd.key' returned 2: New signing request \"etcd\" added.\n<13>Oct 14 13:27:00 puppet-user: Error: /Stage[main]/Tripleo::Certmonger::Etcd/Certmonger_certificate[etcd]: Could not evaluate: Could not get certificate: Server at https://ipa-server.redhat.local/ipa/json denied our request, giving up: 3009 (invalid 'csr': IP address in subjectAltName (172.20.1.93) unreachable from DNS names).\n<13>Oct 14 13:27:00 puppet-user: Notice: /Stage[main]/Tripleo::Certmonger::Etcd/File[/etc/pki/tls/certs/etcd.crt]: Dependency Certmonger_certificate[etcd] has failures: true\n<13>Oct 14 13:27:00 puppet-user: Warning: /Stage[main]/Tripleo::Certmonger::Etcd/File[/etc/pki/tls/certs/etcd.crt]: Skipping because of failed dependencies\n<13>Oct 14 13:27:00 puppet-user: Warning: /Stage[main]/Tripleo::Certmonger::Etcd/File[/etc/pki/tls/private/etcd.key]: Skipping because of failed dependencies\n<13>Oct 14 13:27:00 puppet-user: Notice: /Stage[main]/Tripleo::Certmonger::Ovn_controller/Certmonger_certificate[ovn_controller]/ensure: created\n<13>Oct 14 13:27:01 puppet-user: Notice: /Stage[main]/Tripleo::Certmonger::Ovn_metadata/Certmonger_certificate[ovn_metadata]/ensure: created\n<13>Oct 14 13:27:02 puppet-user: Notice: /Stage[main]/Tripleo::Profile::Base::Database::Mysql::Client/File[/etc/my.cnf.d/tripleo.cnf]/ensure: created\n<13>Oct 14 13:27:02 puppet-user: Notice: /Stage[main]/Tripleo::Profile::Base::Database::Mysql::Client/Augeas[tripleo-mysql-client-conf]/returns: executed successfully\n<13>Oct 14 13:27:02 puppet-user: Notice: /Stage[main]/Tripleo::Firewall/Exec[save ipv4 rules]/returns: executed successfully\n<13>Oct 14 13:27:02 puppet-user: Notice: /Stage[main]/Tripleo::Firewall/Exec[save ipv6 rules]/returns: executed successfully\n<13>Oct 14 13:27:02 puppet-user: Notice: /Stage[main]/Firewall::Linux::Redhat/Service[iptables]/ensure: ensure changed 'stopped' to 'running'\n<13>Oct 14 13:27:03 puppet-user: Notice: /Stage[main]/Firewall::Linux::Redhat/Service[nftables]/ensure: ensure changed 'stopped' to 'running'\n<13>Oct 14 13:27:03 puppet-user: Notice: /Stage[main]/Firewall::Linux::Redhat/Service[ip6tables]/ensure: ensure changed 'stopped' to 'running'\n<13>Oct 14 13:27:03 puppet-user: Notice: /Stage[main]/Tripleo::Trusted_cas/Tripleo::Trusted_ca[satellite-ca]/File[/etc/pki/ca-trust/source/anchors/satellite-ca.pem]/ensure: defined content as '{md5}21f46a959df87b0766a54407ebb0ca1b'\n<13>Oct 14 13:27:04 puppet-user: Notice: /Stage[main]/Tripleo::Trusted_cas/Tripleo::Trusted_ca[satellite-ca]/Exec[trust-ca-satellite-ca]: Triggered 'refresh' from 1 event\n<13>Oct 14 13:27:04 puppet-user: Notice: /Stage[main]/Tripleo::Trusted_cas/Tripleo::Trusted_ca[undercloud-ca]/File[/etc/pki/ca-trust/source/anchors/undercloud-ca.pem]/ensure: defined content as '{md5}7b779addd597150270a78ddb0bd2c154'\n<13>Oct 14 13:27:04 puppet-user: Notice: /Stage[main]/Tripleo::Trusted_cas/Tripleo::Trusted_ca[undercloud-ca]/Exec[trust-ca-undercloud-ca]: Triggered 'refresh' from 1 event\n<13>Oct 14 13:27:04 puppet-user: Notice: /Stage[main]/Tripleo::Profile::Base::Certmonger_user/Tripleo::Certmonger::Httpd[httpd-ctlplane]/Certmonger_certificate[httpd-ctlplane]/ensure: created\n<13>Oct 14 13:27:06 puppet-user: Notice: /Stage[main]/Tripleo::Profile::Base::Certmonger_user/Tripleo::Certmonger::Httpd[httpd-ctlplane]/Certmonger_certificate[httpd-ctlplane]: Triggered 'refresh' from 2 events\n<13>Oct 14 13:27:06 puppet-user: Notice: /Stage[main]/Tripleo::Profile::Base::Certmonger_user/Tripleo::Certmonger::Httpd[httpd-internal_api]/Certmonger_certificate[httpd-internal_api]/ensure: created\n<13>Oct 14 13:27:07 puppet-user: Notice: /Stage[main]/Tripleo::Profile::Base::Certmonger_user/Tripleo::Certmonger::Httpd[httpd-internal_api]/Certmonger_certificate[httpd-internal_api]: Triggered 'refresh' from 2 events\n<13>Oct 14 13:27:07 puppet-user: Notice: /Stage[main]/Tripleo::Profile::Base::Certmonger_user/Tripleo::Certmonger::Httpd[httpd-storage]/Certmonger_certificate[httpd-storage]/ensure: created\n<13>Oct 14 13:27:23 puppet-user: Notice: /Stage[main]/Tripleo::Profile::Base::Certmonger_user/Tripleo::Certmonger::Httpd[httpd-storage]/Certmonger_certificate[httpd-storage]: Triggered 'refresh' from 2 events\n<13>Oct 14 13:27:23 puppet-user: Notice: /Stage[main]/Tripleo::Profile::Base::Certmonger_user/Tripleo::Certmonger::Httpd[httpd-storage_mgmt]/Certmonger_certificate[httpd-storage_mgmt]/ensure: created\n<13>Oct 14 13:27:25 puppet-user: Notice: /Stage[main]/Tripleo::Profile::Base::Certmonger_user/Tripleo::Certmonger::Httpd[httpd-storage_mgmt]/Certmonger_certificate[httpd-storage_mgmt]: Triggered 'refresh' from 2 events\n<13>Oct 14 13:27:25 puppet-user: Notice: /Stage[main]/Tripleo::Profile::Base::Certmonger_user/Tripleo::Certmonger::Libvirt[libvirt-client-cert]/Certmonger_certificate[libvirt-client-cert]/ensure: created\n<13>Oct 14 13:27:26 puppet-user: Notice: /Stage[main]/Tripleo::Profile::Base::Certmonger_user/Tripleo::Certmonger::Libvirt[libvirt-client-cert]/Certmonger_certificate[libvirt-client-cert]: Triggered 'refresh' from 2 events\n<13>Oct 14 13:27:26 puppet-user: Notice: /Stage[main]/Tripleo::Profile::Base::Certmonger_user/Tripleo::Certmonger::Libvirt[libvirt-server-cert]/Certmonger_certificate[libvirt-server-cert]/ensure: created\n<13>Oct 14 13:27:28 puppet-user: Notice: /Stage[main]/Tripleo::Profile::Base::Certmonger_user/Tripleo::Certmonger::Libvirt[libvirt-server-cert]/Certmonger_certificate[libvirt-server-cert]: Triggered 'refresh' from 2 events\n<13>Oct 14 13:27:28 puppet-user: Notice: /Stage[main]/Tripleo::Profile::Base::Certmonger_user/Tripleo::Certmonger::Libvirt_vnc[libvirt-vnc-server-cert]/Certmonger_certificate[libvirt-vnc-server-cert]/ensure: created\n<13>Oct 14 13:27:29 puppet-user: Notice: /Stage[main]/Tripleo::Profile::Base::Certmonger_user/Tripleo::Certmonger::Libvirt_vnc[libvirt-vnc-server-cert]/Certmonger_certificate[libvirt-vnc-server-cert]: Triggered 'refresh' from 2 events\n<13>Oct 14 13:27:29 puppet-user: Notice: /Stage[main]/Tripleo::Profile::Base::Certmonger_user/Tripleo::Certmonger::Libvirt_vnc[libvirt-vnc-server-cert]/File[/etc/pki/libvirt-vnc/server-cert.pem]/mode: mode changed '0600' to '0644'\n<13>Oct 14 13:27:29 puppet-user: Notice: /Stage[main]/Tripleo::Profile::Base::Certmonger_user/Tripleo::Certmonger::Libvirt_vnc[libvirt-vnc-server-cert]/File[/etc/pki/libvirt-vnc/server-key.pem]/group: group changed 'root' to 'qemu'\n<13>Oct 14 13:27:29 puppet-user: Notice: /Stage[main]/Tripleo::Profile::Base::Certmonger_user/Tripleo::Certmonger::Libvirt_vnc[libvirt-vnc-server-cert]/File[/etc/pki/libvirt-vnc/server-key.pem]/mode: mode changed '0600' to '0640'\n<13>Oct 14 13:27:29 puppet-user: Notice: /Stage[main]/Tripleo::Profile::Base::Certmonger_user/Tripleo::Certmonger::Qemu[qemu-nbd-client-cert]/Certmonger_certificate[qemu-nbd-client-cert]/ensure: created\n<13>Oct 14 13:27:31 puppet-user: Notice: /Stage[main]/Tripleo::Profile::Base::Certmonger_user/Tripleo::Certmonger::Qemu[qemu-nbd-client-cert]/Certmonger_certificate[qemu-nbd-client-cert]: Triggered 'refresh' from 1 event\n<13>Oct 14 13:27:31 puppet-user: Notice: /Stage[main]/Tripleo::Profile::Base::Certmonger_user/Tripleo::Certmonger::Qemu[qemu-nbd-client-cert]/File[/etc/pki/libvirt-nbd/client-cert.pem]/mode: mode changed '0600' to '0644'\n<13>Oct 14 13:27:31 puppet-user: Notice: /Stage[main]/Tripleo::Profile::Base::Certmonger_user/Tripleo::Certmonger::Qemu[qemu-nbd-client-cert]/File[/etc/pki/libvirt-nbd/client-key.pem]/group: group changed 'root' to 'qemu'\n<13>Oct 14 13:27:31 puppet-user: Notice: /Stage[main]/Tripleo::Profile::Base::Certmonger_user/Tripleo::Certmonger::Qemu[qemu-nbd-client-cert]/File[/etc/pki/libvirt-nbd/client-key.pem]/mode: mode changed '0600' to '0640'\n<13>Oct 14 13:27:31 puppet-user: Notice: /Stage[main]/Tripleo::Profile::Base::Certmonger_user/Tripleo::Certmonger::Qemu[qemu-server-cert]/Certmonger_certificate[qemu-server-cert]/ensure: created\n<13>Oct 14 13:27:32 puppet-user: Notice: /Stage[main]/Tripleo::Profile::Base::Certmonger_user/Tripleo::Certmonger::Qemu[qemu-server-cert]/Certmonger_certificate[qemu-server-cert]: Triggered 'refresh' from 2 events\n<13>Oct 14 13:27:32 puppet-user: Warning: /Stage[main]/Tripleo::Certmonger::Ca::Crl/Exec[tripleo-ca-crl]: Skipping because of failed dependencies\n<13>Oct 14 13:27:32 puppet-user: Warning: /Stage[main]/Tripleo::Certmonger::Ca::Crl/File[tripleo-ca-crl-file]: Skipping because of failed dependencies\n<13>Oct 14 13:27:32 puppet-user: Warning: /Stage[main]/Tripleo::Certmonger::Ca::Crl/Exec[tripleo-ca-crl-process-command]: Skipping because of failed dependencies\n<13>Oct 14 13:27:32 puppet-user: Warning: /Stage[main]/Tripleo::Certmonger::Ca::Crl/Cron[tripleo-refresh-crl-file]: Skipping because of failed dependencies\n<13>Oct 14 13:27:32 puppet-user: Notice: /Stage[main]/Tripleo::Profile::Base::Certmonger_user/Tripleo::Certmonger::Qemu[qemu-server-cert]/File[/etc/pki/qemu/server-cert.pem]/mode: mode changed '0600' to '0644'\n<13>Oct 14 13:27:32 puppet-user: Notice: /Stage[main]/Tripleo::Profile::Base::Certmonger_user/Tripleo::Certmonger::Qemu[qemu-server-cert]/File[/etc/pki/qemu/server-key.pem]/group: group changed 'root' to 'qemu'\n<13>Oct 14 13:27:32 puppet-user: Notice: /Stage[main]/Tripleo::Profile::Base::Certmonger_user/Tripleo::Certmonger::Qemu[qemu-server-cert]/File[/etc/pki/qemu/server-key.pem]/mode: mode changed '0600' to '0640'\n<13>Oct 14 13:27:32 puppet-user: Notice: /Stage[main]/Ssh::Server::Config/Concat[/etc/ssh/sshd_config]/File[/etc/ssh/sshd_config]/content: content changed '{md5}ab907f2c9cebdbc4089a852d95bb15dd' to '{md5}3534841fdb8db5b58d66600a60bf3759'\n<13>Oct 14 13:27:32 puppet-user: Notice: /Stage[main]/Ssh::Server::Service/Service[sshd]: Triggered 'refresh' from 2 events\n<13>Oct 14 13:27:33 puppet-user: Notice: /Stage[main]/Tripleo::Firewall::Pre/Tripleo::Firewall::Rule[000 accept related established rules]/Firewall[000 accept related established rules ipv4]/ensure: created\n<13>Oct 14 13:27:33 puppet-user: Notice: /Stage[main]/Tripleo::Firewall::Pre/Tripleo::Firewall::Rule[000 accept related established rules]/Firewall[000 accept related established rules ipv6]/ensure: created\n<13>Oct 14 13:27:33 puppet-user: Notice: /Stage[main]/Tripleo::Firewall::Pre/Tripleo::Firewall::Rule[001 accept all icmp]/Firewall[001 accept all icmp ipv4]/ensure: created\n<13>Oct 14 13:27:33 puppet-user: Notice: /Stage[main]/Tripleo::Firewall::Pre/Tripleo::Firewall::Rule[001 accept all icmp]/Firewall[001 accept all icmp ipv6]/ensure: created\n<13>Oct 14 13:27:33 puppet-user: Notice: /Stage[main]/Tripleo::Firewall::Pre/Tripleo::Firewall::Rule[002 accept all to lo interface]/Firewall[002 accept all to lo interface ipv4]/ensure: created\n<13>Oct 14 13:27:33 puppet-user: Notice: /Stage[main]/Tripleo::Firewall::Pre/Tripleo::Firewall::Rule[002 accept all to lo interface]/Firewall[002 accept all to lo interface ipv6]/ensure: created\n<13>Oct 14 13:27:33 puppet-user: Notice: /Stage[main]/Tripleo::Firewall::Pre/Tripleo::Firewall::Rule[004 accept ipv6 dhcpv6]/Firewall[004 accept ipv6 dhcpv6 ipv6]/ensure: created\n<13>Oct 14 13:27:33 puppet-user: Notice: /Stage[main]/Tripleo::Firewall/Tripleo::Firewall::Service_rules[ceph_mgr]/Tripleo::Firewall::Rule[113 ceph_mgr]/Firewall[113 ceph_mgr ipv4]/ensure: created\n<13>Oct 14 13:27:33 puppet-user: Notice: /Stage[main]/Tripleo::Firewall/Tripleo::Firewall::Service_rules[ceph_mgr]/Tripleo::Firewall::Rule[113 ceph_mgr]/Firewall[113 ceph_mgr ipv6]/ensure: created\n<13>Oct 14 13:27:33 puppet-user: Notice: /Stage[main]/Tripleo::Firewall/Tripleo::Firewall::Service_rules[ceph_mon]/Tripleo::Firewall::Rule[110 ceph_mon]/Firewall[110 ceph_mon ipv4]/ensure: created\n<13>Oct 14 13:27:33 puppet-user: Notice: /Stage[main]/Tripleo::Firewall/Tripleo::Firewall::Service_rules[ceph_mon]/Tripleo::Firewall::Rule[110 ceph_mon]/Firewall[110 ceph_mon ipv6]/ensure: created\n<13>Oct 14 13:27:33 puppet-user: Notice: /Stage[main]/Tripleo::Firewall/Tripleo::Firewall::Service_rules[ceph_osd]/Tripleo::Firewall::Rule[111 ceph_osd]/Firewall[111 ceph_osd ipv4]/ensure: created\n<13>Oct 14 13:27:33 puppet-user: Notice: /Stage[main]/Tripleo::Firewall/Tripleo::Firewall::Service_rules[ceph_osd]/Tripleo::Firewall::Rule[111 ceph_osd]/Firewall[111 ceph_osd ipv6]/ensure: created\n<13>Oct 14 13:27:34 puppet-user: Notice: /Stage[main]/Tripleo::Firewall/Tripleo::Firewall::Service_rules[cinder_volume]/Tripleo::Firewall::Rule[120 iscsi initiator]/Firewall[120 iscsi initiator ipv4]/ensure: created\n<13>Oct 14 13:27:34 puppet-user: Notice: /Stage[main]/Tripleo::Firewall/Tripleo::Firewall::Service_rules[cinder_volume]/Tripleo::Firewall::Rule[120 iscsi initiator]/Firewall[120 iscsi initiator ipv6]/ensure: created\n<13>Oct 14 13:27:34 puppet-user: Notice: /Stage[main]/Tripleo::Firewall/Tripleo::Firewall::Service_rules[etcd]/Tripleo::Firewall::Rule[141 etcd]/Firewall[141 etcd ipv4]/ensure: created\n<13>Oct 14 13:27:34 puppet-user: Notice: /Stage[main]/Tripleo::Firewall/Tripleo::Firewall::Service_rules[etcd]/Tripleo::Firewall::Rule[141 etcd]/Firewall[141 etcd ipv6]/ensure: created\n<13>Oct 14 13:27:34 puppet-user: Notice: /Stage[main]/Tripleo::Firewall/Tripleo::Firewall::Service_rules[glance_api_edge]/Tripleo::Firewall::Rule[112 glance_api_edge]/Firewall[112 glance_api_edge ipv4]/ensure: created\n<13>Oct 14 13:27:34 puppet-user: Notice: /Stage[main]/Tripleo::Firewall/Tripleo::Firewall::Service_rules[glance_api_edge]/Tripleo::Firewall::Rule[112 glance_api_edge]/Firewall[112 glance_api_edge ipv6]/ensure: created\n<13>Oct 14 13:27:34 puppet-user: Notice: /Stage[main]/Tripleo::Firewall/Tripleo::Firewall::Service_rules[nova_libvirt]/Tripleo::Firewall::Rule[200 nova_libvirt]/Firewall[200 nova_libvirt ipv4]/ensure: created\n<13>Oct 14 13:27:34 puppet-user: Notice: /Stage[main]/Tripleo::Firewall/Tripleo::Firewall::Service_rules[nova_libvirt]/Tripleo::Firewall::Rule[200 nova_libvirt]/Firewall[200 nova_libvirt ipv6]/ensure: created\n<13>Oct 14 13:27:34 puppet-user: Notice: /Stage[main]/Tripleo::Firewall/Tripleo::Firewall::Service_rules[nova_migration_target]/Tripleo::Firewall::Rule[113 nova_migration_target accept api subnet 172.20.0.0/24]/Firewall[113 nova_migration_target accept api subnet 172.20.0.0/24 ipv4]/ensure: created\n<13>Oct 14 13:27:34 puppet-user: Notice: /Stage[main]/Tripleo::Firewall/Tripleo::Firewall::Service_rules[nova_migration_target]/Tripleo::Firewall::Rule[113 nova_migration_target accept api subnet 172.20.1.0/24]/Firewall[113 nova_migration_target accept api subnet 172.20.1.0/24 ipv4]/ensure: created\n<13>Oct 14 13:27:35 puppet-user: Notice: /Stage[main]/Tripleo::Firewall/Tripleo::Firewall::Service_rules[nova_migration_target]/Tripleo::Firewall::Rule[113 nova_migration_target accept api subnet 172.20.2.0/24]/Firewall[113 nova_migration_target accept api subnet 172.20.2.0/24 ipv4]/ensure: created\n<13>Oct 14 13:27:35 puppet-user: Notice: /Stage[main]/Tripleo::Firewall/Tripleo::Firewall::Service_rules[nova_migration_target]/Tripleo::Firewall::Rule[113 nova_migration_target accept api subnet 172.20.3.0/24]/Firewall[113 nova_migration_target accept api subnet 172.20.3.0/24 ipv4]/ensure: created\n<13>Oct 14 13:27:35 puppet-user: Notice: /Stage[main]/Tripleo::Firewall/Tripleo::Firewall::Service_rules[nova_migration_target]/Tripleo::Firewall::Rule[113 nova_migration_target accept libvirt subnet 172.20.0.0/24]/Firewall[113 nova_migration_target accept libvirt subnet 172.20.0.0/24 ipv4]/ensure: created\n<13>Oct 14 13:27:35 puppet-user: Notice: /Stage[main]/Tripleo::Firewall/Tripleo::Firewall::Service_rules[nova_migration_target]/Tripleo::Firewall::Rule[113 nova_migration_target accept libvirt subnet 172.20.1.0/24]/Firewall[113 nova_migration_target accept libvirt subnet 172.20.1.0/24 ipv4]/ensure: created\n<13>Oct 14 13:27:35 puppet-user: Notice: /Stage[main]/Tripleo::Firewall/Tripleo::Firewall::Service_rules[nova_migration_target]/Tripleo::Firewall::Rule[113 nova_migration_target accept libvirt subnet 172.20.2.0/24]/Firewall[113 nova_migration_target accept libvirt subnet 172.20.2.0/24 ipv4]/ensure: created\n<13>Oct 14 13:27:35 puppet-user: Notice: /Stage[main]/Tripleo::Firewall/Tripleo::Firewall::Service_rules[nova_migration_target]/Tripleo::Firewall::Rule[113 nova_migration_target accept libvirt subnet 172.20.3.0/24]/Firewall[113 nova_migration_target accept libvirt subnet 172.20.3.0/24 ipv4]/ensure: created\n<13>Oct 14 13:27:35 puppet-user: Notice: /Stage[main]/Tripleo::Firewall/Tripleo::Firewall::Service_rules[snmp]/Tripleo::Firewall::Rule[124 snmp 172.16.0.0/24]/Firewall[124 snmp 172.16.0.0/24 ipv4]/ensure: created\n<13>Oct 14 13:27:35 puppet-user: Notice: /Stage[main]/Tripleo::Firewall/Tripleo::Firewall::Service_rules[snmp]/Tripleo::Firewall::Rule[124 snmp 172.16.1.0/24]/Firewall[124 snmp 172.16.1.0/24 ipv4]/ensure: created\n<13>Oct 14 13:27:35 puppet-user: Notice: /Stage[main]/Tripleo::Firewall/Tripleo::Firewall::Service_rules[snmp]/Tripleo::Firewall::Rule[124 snmp 172.16.2.0/24]/Firewall[124 snmp 172.16.2.0/24 ipv4]/ensure: created\n<13>Oct 14 13:27:36 puppet-user: Notice: /Stage[main]/Tripleo::Firewall/Tripleo::Firewall::Service_rules[snmp]/Tripleo::Firewall::Rule[124 snmp 172.16.3.0/24]/Firewall[124 snmp 172.16.3.0/24 ipv4]/ensure: created\n<13>Oct 14 13:27:36 puppet-user: Notice: /Stage[main]/Tripleo::Firewall/Tripleo::Firewall::Service_rules[snmp]/Tripleo::Firewall::Rule[124 snmp 172.16.4.0/24]/Firewall[124 snmp 172.16.4.0/24 ipv4]/ensure: created\n<13>Oct 14 13:27:36 puppet-user: Notice: /Stage[main]/Tripleo::Firewall/Tripleo::Firewall::Service_rules[chrony]/Tripleo::Firewall::Rule[105 ntp]/Firewall[105 ntp ipv4]/ensure: created\n<13>Oct 14 13:27:36 puppet-user: Notice: /Stage[main]/Tripleo::Firewall/Tripleo::Firewall::Service_rules[chrony]/Tripleo::Firewall::Rule[105 ntp]/Firewall[105 ntp ipv6]/ensure: created\n<13>Oct 14 13:27:36 puppet-user: Notice: /Stage[main]/Tripleo::Firewall/Tripleo::Firewall::Service_rules[tripleo_firewall]/Tripleo::Firewall::Rule[003 accept ssh from ctlplane subnet 172.16.0.0/24]/Firewall[003 accept ssh from ctlplane subnet 172.16.0.0/24 ipv4]/ensure: created\n<13>Oct 14 13:27:36 puppet-user: Notice: /Stage[main]/Tripleo::Firewall/Tripleo::Firewall::Service_rules[tripleo_firewall]/Tripleo::Firewall::Rule[003 accept ssh from ctlplane subnet 172.16.1.0/24]/Firewall[003 accept ssh from ctlplane subnet 172.16.1.0/24 ipv4]/ensure: created\n<13>Oct 14 13:27:36 puppet-user: Notice: /Stage[main]/Tripleo::Firewall/Tripleo::Firewall::Service_rules[tripleo_firewall]/Tripleo::Firewall::Rule[003 accept ssh from ctlplane subnet 172.16.2.0/24]/Firewall[003 accept ssh from ctlplane subnet 172.16.2.0/24 ipv4]/ensure: created\n<13>Oct 14 13:27:37 puppet-user: Notice: /Stage[main]/Tripleo::Firewall/Tripleo::Firewall::Service_rules[tripleo_firewall]/Tripleo::Firewall::Rule[003 accept ssh from ctlplane subnet 172.16.3.0/24]/Firewall[003 accept ssh from ctlplane subnet 172.16.3.0/24 ipv4]/ensure: created\n<13>Oct 14 13:27:37 puppet-user: Notice: /Stage[main]/Tripleo::Firewall/Tripleo::Firewall::Service_rules[tripleo_firewall]/Tripleo::Firewall::Rule[003 accept ssh from ctlplane subnet 172.16.4.0/24]/Firewall[003 accept ssh from ctlplane subnet 172.16.4.0/24 ipv4]/ensure: created\n<13>Oct 14 13:27:37 puppet-user: Notice: /Stage[main]/Tripleo::Firewall/Tripleo::Firewall::Service_rules[ovn_controller]/Tripleo::Firewall::Rule[118 neutron vxlan networks]/Firewall[118 neutron vxlan networks ipv4]/ensure: created\n<13>Oct 14 13:27:37 puppet-user: Notice: /Stage[main]/Tripleo::Firewall/Tripleo::Firewall::Service_rules[ovn_controller]/Tripleo::Firewall::Rule[118 neutron vxlan networks]/Firewall[118 neutron vxlan networks ipv6]/ensure: created\n<13>Oct 14 13:27:37 puppet-user: Notice: /Stage[main]/Tripleo::Firewall/Tripleo::Firewall::Service_rules[ovn_controller]/Tripleo::Firewall::Rule[119 neutron geneve networks]/Firewall[119 neutron geneve networks ipv4]/ensure: created\n<13>Oct 14 13:27:37 puppet-user: Notice: /Stage[main]/Tripleo::Firewall/Tripleo::Firewall::Service_rules[ovn_controller]/Tripleo::Firewall::Rule[119 neutron geneve networks]/Firewall[119 neutron geneve networks ipv6]/ensure: created\n<13>Oct 14 13:27:37 puppet-user: Notice: /Stage[main]/Tripleo::Firewall::Post/Tripleo::Firewall::Rule[998 log all]/Firewall[998 log all ipv4]/ensure: created\n<13>Oct 14 13:27:38 puppet-user: Notice: /Stage[main]/Tripleo::Firewall::Post/Tripleo::Firewall::Rule[998 log all]/Firewall[998 log all ipv6]/ensure: created\n<13>Oct 14 13:27:38 puppet-user: Notice: /Stage[main]/Tripleo::Firewall::Post/Tripleo::Firewall::Rule[999 drop all]/Firewall[999 drop all ipv4]/ensure: created\n<13>Oct 14 13:27:38 puppet-user: Notice: /Stage[main]/Tripleo::Firewall::Post/Tripleo::Firewall::Rule[999 drop all]/Firewall[999 drop all ipv6]/ensure: created\n<13>Oct 14 13:27:38 puppet-user: Notice: Applied catalog in 39.45 seconds\n<13>Oct 14 13:27:38 puppet-user: Changes:\n<13>Oct 14 13:27:38 puppet-user:             Total: 105\n<13>Oct 14 13:27:38 puppet-user: Events:\n<13>Oct 14 13:27:38 puppet-user:           Failure: 1\n<13>Oct 14 13:27:38 puppet-user:           Success: 105\n<13>Oct 14 13:27:38 puppet-user:             Total: 106\n<13>Oct 14 13:27:38 puppet-user: Resources:\n<13>Oct 14 13:27:38 puppet-user:            Failed: 1\n<13>Oct 14 13:27:38 puppet-user:         Restarted: 13\n<13>Oct 14 13:27:38 puppet-user:           Skipped: 6\n<13>Oct 14 13:27:38 puppet-user:           Changed: 92\n<13>Oct 14 13:27:38 puppet-user:       Out of sync: 92\n<13>Oct 14 13:27:38 puppet-user:             Total: 131\n<13>Oct 14 13:27:38 puppet-user: Time:\n<13>Oct 14 13:27:38 puppet-user:            Anchor: 0.00\n<13>Oct 14 13:27:38 puppet-user:          Schedule: 0.00\n<13>Oct 14 13:27:38 puppet-user:    Package manifest: 0.00\n<13>Oct 14 13:27:38 puppet-user:            Augeas: 0.02\n<13>Oct 14 13:27:38 puppet-user:           Package: 0.07\n<13>Oct 14 13:27:38 puppet-user:              Exec: 0.09\n<13>Oct 14 13:27:38 puppet-user:              File: 0.14\n<13>Oct 14 13:27:38 puppet-user:    Config retrieval: 1.25\n<13>Oct 14 13:27:38 puppet-user:          Last run: 1634218058\n<13>Oct 14 13:27:38 puppet-user:           Service: 2.06\n<13>Oct 14 13:27:38 puppet-user:    Transaction evaluation: 39.43\n<13>Oct 14 13:27:38 puppet-user:    Catalog application: 39.45\n<13>Oct 14 13:27:38 puppet-user:        Filebucket: 0.00\n<13>Oct 14 13:27:38 puppet-user:          Firewall: 5.26\n<13>Oct 14 13:27:38 puppet-user:    Concat fragment: 0.00\n<13>Oct 14 13:27:38 puppet-user:       Concat file: 0.00\n<13>Oct 14 13:27:38 puppet-user:    Certmonger certificate: 9.63\n<13>Oct 14 13:27:38 puppet-user:             Total: 39.45\n<13>Oct 14 13:27:38 puppet-user: Version:\n<13>Oct 14 13:27:38 puppet-user:            Config: 1634218017\n<13>Oct 14 13:27:38 puppet-user:            Puppet: 5.5.10", "stderr_lines": ["<13>Oct 14 13:26:52 puppet-user: Warning: The function 'hiera' is deprecated in favor of using 'lookup'. See https://puppet.com/docs/puppet/5.5/deprecated_language.html\\n   (file & line not available)", "<13>Oct 14 13:26:57 puppet-user: Warning: /etc/puppet/hiera.yaml: Use of 'hiera.yaml' version 3 is deprecated. It should be converted to version 5", "<13>Oct 14 13:26:57 puppet-user:    (file: /etc/puppet/hiera.yaml)", "<13>Oct 14 13:26:57 puppet-user: Warning: Undefined variable '::deploy_config_name'; \\n   (file & line not available)", "<13>Oct 14 13:26:57 puppet-user: Warning: ModuleLoader: module 'tripleo' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules\\n   (file & line not available)", "<13>Oct 14 13:26:58 puppet-user: Warning: Unknown variable: '::deployment_type'. (file: /etc/puppet/modules/tripleo/manifests/profile/base/database/mysql/client.pp, line: 89, column: 8)", "<13>Oct 14 13:26:58 puppet-user: Warning: ModuleLoader: module 'concat' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules\\n   (file & line not available)", "<13>Oct 14 13:26:58 puppet-user: Warning: tag is a metaparam; this value will inherit to all contained resources in the tripleo::firewall::rule definition", "<13>Oct 14 13:26:58 puppet-user: Notice: Scope(Class[Tripleo::Firewall::Post]): At this stage, all network traffic is blocked.", "<13>Oct 14 13:26:58 puppet-user: Warning: ModuleLoader: module 'nova' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules\\n   (file & line not available)", "<13>Oct 14 13:26:58 puppet-user: Warning: ModuleLoader: module 'openstacklib' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules\\n   (file & line not available)", "<13>Oct 14 13:26:58 puppet-user: Notice: Compiled catalog for hci-3-edge-1.redhat.local in environment production in 0.96 seconds", "<13>Oct 14 13:26:59 puppet-user: Notice: /Stage[main]/Main/Package_manifest[/var/lib/tripleo/installed-packages/overcloud_HCIEdge11]/ensure: created", "<13>Oct 14 13:26:59 puppet-user: Notice: /Stage[main]/Tripleo::Certmonger::Apache_dirs/File[/etc/pki/tls/certs/httpd]/ensure: created", "<13>Oct 14 13:26:59 puppet-user: Notice: /Stage[main]/Tripleo::Certmonger::Apache_dirs/File[/etc/pki/tls/private/httpd]/ensure: created", "<13>Oct 14 13:26:59 puppet-user: Notice: /Stage[main]/Tripleo::Certmonger::Libvirt_dirs/File[/etc/pki/libvirt]/ensure: created", "<13>Oct 14 13:26:59 puppet-user: Notice: /Stage[main]/Tripleo::Certmonger::Libvirt_dirs/File[/etc/pki/libvirt/private]/ensure: created", "<13>Oct 14 13:26:59 puppet-user: Notice: /Stage[main]/Tripleo::Certmonger::Libvirt_vnc_dirs/File[/etc/pki/libvirt-vnc]/ensure: created", "<13>Oct 14 13:26:59 puppet-user: Notice: /Stage[main]/Tripleo::Certmonger::Ca::Libvirt_vnc/File[/etc/pki/libvirt-vnc/ca-cert.pem]/ensure: created", "<13>Oct 14 13:26:59 puppet-user: Notice: /Stage[main]/Tripleo::Certmonger::Qemu_dirs/File[/etc/pki/qemu]/ensure: created", "<13>Oct 14 13:26:59 puppet-user: Notice: /Stage[main]/Tripleo::Certmonger::Ca::Qemu/File[/etc/pki/qemu/ca-cert.pem]/ensure: created", "<13>Oct 14 13:26:59 puppet-user: Notice: /Stage[main]/Tripleo::Certmonger::Qemu_nbd_dirs/File[/etc/pki/libvirt-nbd]/ensure: created", "<13>Oct 14 13:26:59 puppet-user: Notice: /Stage[main]/Tripleo::Certmonger::Etcd/File[/usr/bin/certmonger-etcd-refresh.sh]/ensure: defined content as '{md5}0aa1a0196e0573519b00c15035ea4947'", "<13>Oct 14 13:26:59 puppet-user: Notice: /Stage[main]/Certmonger/Service[certmonger]/enable: enable changed 'false' to 'true'", "<13>Oct 14 13:26:59 puppet-user: Notice: /Stage[main]/Certmonger/Service[certmonger]: Triggered 'refresh' from 1 event", "<13>Oct 14 13:26:59 puppet-user: Notice: /Stage[main]/Tripleo::Certmonger::Etcd/Certmonger_certificate[etcd]/ensure: created", "<13>Oct 14 13:26:59 puppet-user: Warning: Could not get certificate: Execution of '/usr/bin/getcert request -I etcd -f /etc/pki/tls/certs/etcd.crt -c IPA -N CN=hci-3-edge-1.internalapi.redhat.local -K etcd/hci-3-edge-1.internalapi.redhat.local -D hci-3-edge-1.internalapi.redhat.local -A 172.20.1.93 -C /usr/bin/certmonger-etcd-refresh.sh -g 2048 -w -k /etc/pki/tls/private/etcd.key' returned 2: New signing request \"etcd\" added.", "<13>Oct 14 13:27:00 puppet-user: Error: /Stage[main]/Tripleo::Certmonger::Etcd/Certmonger_certificate[etcd]: Could not evaluate: Could not get certificate: Server at https://ipa-server.redhat.local/ipa/json denied our request, giving up: 3009 (invalid 'csr': IP address in subjectAltName (172.20.1.93) unreachable from DNS names).", "<13>Oct 14 13:27:00 puppet-user: Notice: /Stage[main]/Tripleo::Certmonger::Etcd/File[/etc/pki/tls/certs/etcd.crt]: Dependency Certmonger_certificate[etcd] has failures: true", "<13>Oct 14 13:27:00 puppet-user: Warning: /Stage[main]/Tripleo::Certmonger::Etcd/File[/etc/pki/tls/certs/etcd.crt]: Skipping because of failed dependencies", "<13>Oct 14 13:27:00 puppet-user: Warning: /Stage[main]/Tripleo::Certmonger::Etcd/File[/etc/pki/tls/private/etcd.key]: Skipping because of failed dependencies", "<13>Oct 14 13:27:00 puppet-user: Notice: /Stage[main]/Tripleo::Certmonger::Ovn_controller/Certmonger_certificate[ovn_controller]/ensure: created", "<13>Oct 14 13:27:01 puppet-user: Notice: /Stage[main]/Tripleo::Certmonger::Ovn_metadata/Certmonger_certificate[ovn_metadata]/ensure: created", "<13>Oct 14 13:27:02 puppet-user: Notice: /Stage[main]/Tripleo::Profile::Base::Database::Mysql::Client/File[/etc/my.cnf.d/tripleo.cnf]/ensure: created", "<13>Oct 14 13:27:02 puppet-user: Notice: /Stage[main]/Tripleo::Profile::Base::Database::Mysql::Client/Augeas[tripleo-mysql-client-conf]/returns: executed successfully", "<13>Oct 14 13:27:02 puppet-user: Notice: /Stage[main]/Tripleo::Firewall/Exec[save ipv4 rules]/returns: executed successfully", "<13>Oct 14 13:27:02 puppet-user: Notice: /Stage[main]/Tripleo::Firewall/Exec[save ipv6 rules]/returns: executed successfully", "<13>Oct 14 13:27:02 puppet-user: Notice: /Stage[main]/Firewall::Linux::Redhat/Service[iptables]/ensure: ensure changed 'stopped' to 'running'", "<13>Oct 14 13:27:03 puppet-user: Notice: /Stage[main]/Firewall::Linux::Redhat/Service[nftables]/ensure: ensure changed 'stopped' to 'running'", "<13>Oct 14 13:27:03 puppet-user: Notice: /Stage[main]/Firewall::Linux::Redhat/Service[ip6tables]/ensure: ensure changed 'stopped' to 'running'", "<13>Oct 14 13:27:03 puppet-user: Notice: /Stage[main]/Tripleo::Trusted_cas/Tripleo::Trusted_ca[satellite-ca]/File[/etc/pki/ca-trust/source/anchors/satellite-ca.pem]/ensure: defined content as '{md5}21f46a959df87b0766a54407ebb0ca1b'", "<13>Oct 14 13:27:04 puppet-user: Notice: /Stage[main]/Tripleo::Trusted_cas/Tripleo::Trusted_ca[satellite-ca]/Exec[trust-ca-satellite-ca]: Triggered 'refresh' from 1 event", "<13>Oct 14 13:27:04 puppet-user: Notice: /Stage[main]/Tripleo::Trusted_cas/Tripleo::Trusted_ca[undercloud-ca]/File[/etc/pki/ca-trust/source/anchors/undercloud-ca.pem]/ensure: defined content as '{md5}7b779addd597150270a78ddb0bd2c154'", "<13>Oct 14 13:27:04 puppet-user: Notice: /Stage[main]/Tripleo::Trusted_cas/Tripleo::Trusted_ca[undercloud-ca]/Exec[trust-ca-undercloud-ca]: Triggered 'refresh' from 1 event", "<13>Oct 14 13:27:04 puppet-user: Notice: /Stage[main]/Tripleo::Profile::Base::Certmonger_user/Tripleo::Certmonger::Httpd[httpd-ctlplane]/Certmonger_certificate[httpd-ctlplane]/ensure: created", "<13>Oct 14 13:27:06 puppet-user: Notice: /Stage[main]/Tripleo::Profile::Base::Certmonger_user/Tripleo::Certmonger::Httpd[httpd-ctlplane]/Certmonger_certificate[httpd-ctlplane]: Triggered 'refresh' from 2 events", "<13>Oct 14 13:27:06 puppet-user: Notice: /Stage[main]/Tripleo::Profile::Base::Certmonger_user/Tripleo::Certmonger::Httpd[httpd-internal_api]/Certmonger_certificate[httpd-internal_api]/ensure: created", "<13>Oct 14 13:27:07 puppet-user: Notice: /Stage[main]/Tripleo::Profile::Base::Certmonger_user/Tripleo::Certmonger::Httpd[httpd-internal_api]/Certmonger_certificate[httpd-internal_api]: Triggered 'refresh' from 2 events", "<13>Oct 14 13:27:07 puppet-user: Notice: /Stage[main]/Tripleo::Profile::Base::Certmonger_user/Tripleo::Certmonger::Httpd[httpd-storage]/Certmonger_certificate[httpd-storage]/ensure: created", "<13>Oct 14 13:27:23 puppet-user: Notice: /Stage[main]/Tripleo::Profile::Base::Certmonger_user/Tripleo::Certmonger::Httpd[httpd-storage]/Certmonger_certificate[httpd-storage]: Triggered 'refresh' from 2 events", "<13>Oct 14 13:27:23 puppet-user: Notice: /Stage[main]/Tripleo::Profile::Base::Certmonger_user/Tripleo::Certmonger::Httpd[httpd-storage_mgmt]/Certmonger_certificate[httpd-storage_mgmt]/ensure: created", "<13>Oct 14 13:27:25 puppet-user: Notice: /Stage[main]/Tripleo::Profile::Base::Certmonger_user/Tripleo::Certmonger::Httpd[httpd-storage_mgmt]/Certmonger_certificate[httpd-storage_mgmt]: Triggered 'refresh' from 2 events", "<13>Oct 14 13:27:25 puppet-user: Notice: /Stage[main]/Tripleo::Profile::Base::Certmonger_user/Tripleo::Certmonger::Libvirt[libvirt-client-cert]/Certmonger_certificate[libvirt-client-cert]/ensure: created", "<13>Oct 14 13:27:26 puppet-user: Notice: /Stage[main]/Tripleo::Profile::Base::Certmonger_user/Tripleo::Certmonger::Libvirt[libvirt-client-cert]/Certmonger_certificate[libvirt-client-cert]: Triggered 'refresh' from 2 events", "<13>Oct 14 13:27:26 puppet-user: Notice: /Stage[main]/Tripleo::Profile::Base::Certmonger_user/Tripleo::Certmonger::Libvirt[libvirt-server-cert]/Certmonger_certificate[libvirt-server-cert]/ensure: created", "<13>Oct 14 13:27:28 puppet-user: Notice: /Stage[main]/Tripleo::Profile::Base::Certmonger_user/Tripleo::Certmonger::Libvirt[libvirt-server-cert]/Certmonger_certificate[libvirt-server-cert]: Triggered 'refresh' from 2 events", "<13>Oct 14 13:27:28 puppet-user: Notice: /Stage[main]/Tripleo::Profile::Base::Certmonger_user/Tripleo::Certmonger::Libvirt_vnc[libvirt-vnc-server-cert]/Certmonger_certificate[libvirt-vnc-server-cert]/ensure: created", "<13>Oct 14 13:27:29 puppet-user: Notice: /Stage[main]/Tripleo::Profile::Base::Certmonger_user/Tripleo::Certmonger::Libvirt_vnc[libvirt-vnc-server-cert]/Certmonger_certificate[libvirt-vnc-server-cert]: Triggered 'refresh' from 2 events", "<13>Oct 14 13:27:29 puppet-user: Notice: /Stage[main]/Tripleo::Profile::Base::Certmonger_user/Tripleo::Certmonger::Libvirt_vnc[libvirt-vnc-server-cert]/File[/etc/pki/libvirt-vnc/server-cert.pem]/mode: mode changed '0600' to '0644'", "<13>Oct 14 13:27:29 puppet-user: Notice: /Stage[main]/Tripleo::Profile::Base::Certmonger_user/Tripleo::Certmonger::Libvirt_vnc[libvirt-vnc-server-cert]/File[/etc/pki/libvirt-vnc/server-key.pem]/group: group changed 'root' to 'qemu'", "<13>Oct 14 13:27:29 puppet-user: Notice: /Stage[main]/Tripleo::Profile::Base::Certmonger_user/Tripleo::Certmonger::Libvirt_vnc[libvirt-vnc-server-cert]/File[/etc/pki/libvirt-vnc/server-key.pem]/mode: mode changed '0600' to '0640'", "<13>Oct 14 13:27:29 puppet-user: Notice: /Stage[main]/Tripleo::Profile::Base::Certmonger_user/Tripleo::Certmonger::Qemu[qemu-nbd-client-cert]/Certmonger_certificate[qemu-nbd-client-cert]/ensure: created", "<13>Oct 14 13:27:31 puppet-user: Notice: /Stage[main]/Tripleo::Profile::Base::Certmonger_user/Tripleo::Certmonger::Qemu[qemu-nbd-client-cert]/Certmonger_certificate[qemu-nbd-client-cert]: Triggered 'refresh' from 1 event", "<13>Oct 14 13:27:31 puppet-user: Notice: /Stage[main]/Tripleo::Profile::Base::Certmonger_user/Tripleo::Certmonger::Qemu[qemu-nbd-client-cert]/File[/etc/pki/libvirt-nbd/client-cert.pem]/mode: mode changed '0600' to '0644'", "<13>Oct 14 13:27:31 puppet-user: Notice: /Stage[main]/Tripleo::Profile::Base::Certmonger_user/Tripleo::Certmonger::Qemu[qemu-nbd-client-cert]/File[/etc/pki/libvirt-nbd/client-key.pem]/group: group changed 'root' to 'qemu'", "<13>Oct 14 13:27:31 puppet-user: Notice: /Stage[main]/Tripleo::Profile::Base::Certmonger_user/Tripleo::Certmonger::Qemu[qemu-nbd-client-cert]/File[/etc/pki/libvirt-nbd/client-key.pem]/mode: mode changed '0600' to '0640'", "<13>Oct 14 13:27:31 puppet-user: Notice: /Stage[main]/Tripleo::Profile::Base::Certmonger_user/Tripleo::Certmonger::Qemu[qemu-server-cert]/Certmonger_certificate[qemu-server-cert]/ensure: created", "<13>Oct 14 13:27:32 puppet-user: Notice: /Stage[main]/Tripleo::Profile::Base::Certmonger_user/Tripleo::Certmonger::Qemu[qemu-server-cert]/Certmonger_certificate[qemu-server-cert]: Triggered 'refresh' from 2 events", "<13>Oct 14 13:27:32 puppet-user: Warning: /Stage[main]/Tripleo::Certmonger::Ca::Crl/Exec[tripleo-ca-crl]: Skipping because of failed dependencies", "<13>Oct 14 13:27:32 puppet-user: Warning: /Stage[main]/Tripleo::Certmonger::Ca::Crl/File[tripleo-ca-crl-file]: Skipping because of failed dependencies", "<13>Oct 14 13:27:32 puppet-user: Warning: /Stage[main]/Tripleo::Certmonger::Ca::Crl/Exec[tripleo-ca-crl-process-command]: Skipping because of failed dependencies", "<13>Oct 14 13:27:32 puppet-user: Warning: /Stage[main]/Tripleo::Certmonger::Ca::Crl/Cron[tripleo-refresh-crl-file]: Skipping because of failed dependencies", "<13>Oct 14 13:27:32 puppet-user: Notice: /Stage[main]/Tripleo::Profile::Base::Certmonger_user/Tripleo::Certmonger::Qemu[qemu-server-cert]/File[/etc/pki/qemu/server-cert.pem]/mode: mode changed '0600' to '0644'", "<13>Oct 14 13:27:32 puppet-user: Notice: /Stage[main]/Tripleo::Profile::Base::Certmonger_user/Tripleo::Certmonger::Qemu[qemu-server-cert]/File[/etc/pki/qemu/server-key.pem]/group: group changed 'root' to 'qemu'", "<13>Oct 14 13:27:32 puppet-user: Notice: /Stage[main]/Tripleo::Profile::Base::Certmonger_user/Tripleo::Certmonger::Qemu[qemu-server-cert]/File[/etc/pki/qemu/server-key.pem]/mode: mode changed '0600' to '0640'", "<13>Oct 14 13:27:32 puppet-user: Notice: /Stage[main]/Ssh::Server::Config/Concat[/etc/ssh/sshd_config]/File[/etc/ssh/sshd_config]/content: content changed '{md5}ab907f2c9cebdbc4089a852d95bb15dd' to '{md5}3534841fdb8db5b58d66600a60bf3759'", "<13>Oct 14 13:27:32 puppet-user: Notice: /Stage[main]/Ssh::Server::Service/Service[sshd]: Triggered 'refresh' from 2 events", "<13>Oct 14 13:27:33 puppet-user: Notice: /Stage[main]/Tripleo::Firewall::Pre/Tripleo::Firewall::Rule[000 accept related established rules]/Firewall[000 accept related established rules ipv4]/ensure: created", "<13>Oct 14 13:27:33 puppet-user: Notice: /Stage[main]/Tripleo::Firewall::Pre/Tripleo::Firewall::Rule[000 accept related established rules]/Firewall[000 accept related established rules ipv6]/ensure: created", "<13>Oct 14 13:27:33 puppet-user: Notice: /Stage[main]/Tripleo::Firewall::Pre/Tripleo::Firewall::Rule[001 accept all icmp]/Firewall[001 accept all icmp ipv4]/ensure: created", "<13>Oct 14 13:27:33 puppet-user: Notice: /Stage[main]/Tripleo::Firewall::Pre/Tripleo::Firewall::Rule[001 accept all icmp]/Firewall[001 accept all icmp ipv6]/ensure: created", "<13>Oct 14 13:27:33 puppet-user: Notice: /Stage[main]/Tripleo::Firewall::Pre/Tripleo::Firewall::Rule[002 accept all to lo interface]/Firewall[002 accept all to lo interface ipv4]/ensure: created", "<13>Oct 14 13:27:33 puppet-user: Notice: /Stage[main]/Tripleo::Firewall::Pre/Tripleo::Firewall::Rule[002 accept all to lo interface]/Firewall[002 accept all to lo interface ipv6]/ensure: created", "<13>Oct 14 13:27:33 puppet-user: Notice: /Stage[main]/Tripleo::Firewall::Pre/Tripleo::Firewall::Rule[004 accept ipv6 dhcpv6]/Firewall[004 accept ipv6 dhcpv6 ipv6]/ensure: created", "<13>Oct 14 13:27:33 puppet-user: Notice: /Stage[main]/Tripleo::Firewall/Tripleo::Firewall::Service_rules[ceph_mgr]/Tripleo::Firewall::Rule[113 ceph_mgr]/Firewall[113 ceph_mgr ipv4]/ensure: created", "<13>Oct 14 13:27:33 puppet-user: Notice: /Stage[main]/Tripleo::Firewall/Tripleo::Firewall::Service_rules[ceph_mgr]/Tripleo::Firewall::Rule[113 ceph_mgr]/Firewall[113 ceph_mgr ipv6]/ensure: created", "<13>Oct 14 13:27:33 puppet-user: Notice: /Stage[main]/Tripleo::Firewall/Tripleo::Firewall::Service_rules[ceph_mon]/Tripleo::Firewall::Rule[110 ceph_mon]/Firewall[110 ceph_mon ipv4]/ensure: created", "<13>Oct 14 13:27:33 puppet-user: Notice: /Stage[main]/Tripleo::Firewall/Tripleo::Firewall::Service_rules[ceph_mon]/Tripleo::Firewall::Rule[110 ceph_mon]/Firewall[110 ceph_mon ipv6]/ensure: created", "<13>Oct 14 13:27:33 puppet-user: Notice: /Stage[main]/Tripleo::Firewall/Tripleo::Firewall::Service_rules[ceph_osd]/Tripleo::Firewall::Rule[111 ceph_osd]/Firewall[111 ceph_osd ipv4]/ensure: created", "<13>Oct 14 13:27:33 puppet-user: Notice: /Stage[main]/Tripleo::Firewall/Tripleo::Firewall::Service_rules[ceph_osd]/Tripleo::Firewall::Rule[111 ceph_osd]/Firewall[111 ceph_osd ipv6]/ensure: created", "<13>Oct 14 13:27:34 puppet-user: Notice: /Stage[main]/Tripleo::Firewall/Tripleo::Firewall::Service_rules[cinder_volume]/Tripleo::Firewall::Rule[120 iscsi initiator]/Firewall[120 iscsi initiator ipv4]/ensure: created", "<13>Oct 14 13:27:34 puppet-user: Notice: /Stage[main]/Tripleo::Firewall/Tripleo::Firewall::Service_rules[cinder_volume]/Tripleo::Firewall::Rule[120 iscsi initiator]/Firewall[120 iscsi initiator ipv6]/ensure: created", "<13>Oct 14 13:27:34 puppet-user: Notice: /Stage[main]/Tripleo::Firewall/Tripleo::Firewall::Service_rules[etcd]/Tripleo::Firewall::Rule[141 etcd]/Firewall[141 etcd ipv4]/ensure: created", "<13>Oct 14 13:27:34 puppet-user: Notice: /Stage[main]/Tripleo::Firewall/Tripleo::Firewall::Service_rules[etcd]/Tripleo::Firewall::Rule[141 etcd]/Firewall[141 etcd ipv6]/ensure: created", "<13>Oct 14 13:27:34 puppet-user: Notice: /Stage[main]/Tripleo::Firewall/Tripleo::Firewall::Service_rules[glance_api_edge]/Tripleo::Firewall::Rule[112 glance_api_edge]/Firewall[112 glance_api_edge ipv4]/ensure: created", "<13>Oct 14 13:27:34 puppet-user: Notice: /Stage[main]/Tripleo::Firewall/Tripleo::Firewall::Service_rules[glance_api_edge]/Tripleo::Firewall::Rule[112 glance_api_edge]/Firewall[112 glance_api_edge ipv6]/ensure: created", "<13>Oct 14 13:27:34 puppet-user: Notice: /Stage[main]/Tripleo::Firewall/Tripleo::Firewall::Service_rules[nova_libvirt]/Tripleo::Firewall::Rule[200 nova_libvirt]/Firewall[200 nova_libvirt ipv4]/ensure: created", "<13>Oct 14 13:27:34 puppet-user: Notice: /Stage[main]/Tripleo::Firewall/Tripleo::Firewall::Service_rules[nova_libvirt]/Tripleo::Firewall::Rule[200 nova_libvirt]/Firewall[200 nova_libvirt ipv6]/ensure: created", "<13>Oct 14 13:27:34 puppet-user: Notice: /Stage[main]/Tripleo::Firewall/Tripleo::Firewall::Service_rules[nova_migration_target]/Tripleo::Firewall::Rule[113 nova_migration_target accept api subnet 172.20.0.0/24]/Firewall[113 nova_migration_target accept api subnet 172.20.0.0/24 ipv4]/ensure: created", "<13>Oct 14 13:27:34 puppet-user: Notice: /Stage[main]/Tripleo::Firewall/Tripleo::Firewall::Service_rules[nova_migration_target]/Tripleo::Firewall::Rule[113 nova_migration_target accept api subnet 172.20.1.0/24]/Firewall[113 nova_migration_target accept api subnet 172.20.1.0/24 ipv4]/ensure: created", "<13>Oct 14 13:27:35 puppet-user: Notice: /Stage[main]/Tripleo::Firewall/Tripleo::Firewall::Service_rules[nova_migration_target]/Tripleo::Firewall::Rule[113 nova_migration_target accept api subnet 172.20.2.0/24]/Firewall[113 nova_migration_target accept api subnet 172.20.2.0/24 ipv4]/ensure: created", "<13>Oct 14 13:27:35 puppet-user: Notice: /Stage[main]/Tripleo::Firewall/Tripleo::Firewall::Service_rules[nova_migration_target]/Tripleo::Firewall::Rule[113 nova_migration_target accept api subnet 172.20.3.0/24]/Firewall[113 nova_migration_target accept api subnet 172.20.3.0/24 ipv4]/ensure: created", "<13>Oct 14 13:27:35 puppet-user: Notice: /Stage[main]/Tripleo::Firewall/Tripleo::Firewall::Service_rules[nova_migration_target]/Tripleo::Firewall::Rule[113 nova_migration_target accept libvirt subnet 172.20.0.0/24]/Firewall[113 nova_migration_target accept libvirt subnet 172.20.0.0/24 ipv4]/ensure: created", "<13>Oct 14 13:27:35 puppet-user: Notice: /Stage[main]/Tripleo::Firewall/Tripleo::Firewall::Service_rules[nova_migration_target]/Tripleo::Firewall::Rule[113 nova_migration_target accept libvirt subnet 172.20.1.0/24]/Firewall[113 nova_migration_target accept libvirt subnet 172.20.1.0/24 ipv4]/ensure: created", "<13>Oct 14 13:27:35 puppet-user: Notice: /Stage[main]/Tripleo::Firewall/Tripleo::Firewall::Service_rules[nova_migration_target]/Tripleo::Firewall::Rule[113 nova_migration_target accept libvirt subnet 172.20.2.0/24]/Firewall[113 nova_migration_target accept libvirt subnet 172.20.2.0/24 ipv4]/ensure: created", "<13>Oct 14 13:27:35 puppet-user: Notice: /Stage[main]/Tripleo::Firewall/Tripleo::Firewall::Service_rules[nova_migration_target]/Tripleo::Firewall::Rule[113 nova_migration_target accept libvirt subnet 172.20.3.0/24]/Firewall[113 nova_migration_target accept libvirt subnet 172.20.3.0/24 ipv4]/ensure: created", "<13>Oct 14 13:27:35 puppet-user: Notice: /Stage[main]/Tripleo::Firewall/Tripleo::Firewall::Service_rules[snmp]/Tripleo::Firewall::Rule[124 snmp 172.16.0.0/24]/Firewall[124 snmp 172.16.0.0/24 ipv4]/ensure: created", "<13>Oct 14 13:27:35 puppet-user: Notice: /Stage[main]/Tripleo::Firewall/Tripleo::Firewall::Service_rules[snmp]/Tripleo::Firewall::Rule[124 snmp 172.16.1.0/24]/Firewall[124 snmp 172.16.1.0/24 ipv4]/ensure: created", "<13>Oct 14 13:27:35 puppet-user: Notice: /Stage[main]/Tripleo::Firewall/Tripleo::Firewall::Service_rules[snmp]/Tripleo::Firewall::Rule[124 snmp 172.16.2.0/24]/Firewall[124 snmp 172.16.2.0/24 ipv4]/ensure: created", "<13>Oct 14 13:27:36 puppet-user: Notice: /Stage[main]/Tripleo::Firewall/Tripleo::Firewall::Service_rules[snmp]/Tripleo::Firewall::Rule[124 snmp 172.16.3.0/24]/Firewall[124 snmp 172.16.3.0/24 ipv4]/ensure: created", "<13>Oct 14 13:27:36 puppet-user: Notice: /Stage[main]/Tripleo::Firewall/Tripleo::Firewall::Service_rules[snmp]/Tripleo::Firewall::Rule[124 snmp 172.16.4.0/24]/Firewall[124 snmp 172.16.4.0/24 ipv4]/ensure: created", "<13>Oct 14 13:27:36 puppet-user: Notice: /Stage[main]/Tripleo::Firewall/Tripleo::Firewall::Service_rules[chrony]/Tripleo::Firewall::Rule[105 ntp]/Firewall[105 ntp ipv4]/ensure: created", "<13>Oct 14 13:27:36 puppet-user: Notice: /Stage[main]/Tripleo::Firewall/Tripleo::Firewall::Service_rules[chrony]/Tripleo::Firewall::Rule[105 ntp]/Firewall[105 ntp ipv6]/ensure: created", "<13>Oct 14 13:27:36 puppet-user: Notice: /Stage[main]/Tripleo::Firewall/Tripleo::Firewall::Service_rules[tripleo_firewall]/Tripleo::Firewall::Rule[003 accept ssh from ctlplane subnet 172.16.0.0/24]/Firewall[003 accept ssh from ctlplane subnet 172.16.0.0/24 ipv4]/ensure: created", "<13>Oct 14 13:27:36 puppet-user: Notice: /Stage[main]/Tripleo::Firewall/Tripleo::Firewall::Service_rules[tripleo_firewall]/Tripleo::Firewall::Rule[003 accept ssh from ctlplane subnet 172.16.1.0/24]/Firewall[003 accept ssh from ctlplane subnet 172.16.1.0/24 ipv4]/ensure: created", "<13>Oct 14 13:27:36 puppet-user: Notice: /Stage[main]/Tripleo::Firewall/Tripleo::Firewall::Service_rules[tripleo_firewall]/Tripleo::Firewall::Rule[003 accept ssh from ctlplane subnet 172.16.2.0/24]/Firewall[003 accept ssh from ctlplane subnet 172.16.2.0/24 ipv4]/ensure: created", "<13>Oct 14 13:27:37 puppet-user: Notice: /Stage[main]/Tripleo::Firewall/Tripleo::Firewall::Service_rules[tripleo_firewall]/Tripleo::Firewall::Rule[003 accept ssh from ctlplane subnet 172.16.3.0/24]/Firewall[003 accept ssh from ctlplane subnet 172.16.3.0/24 ipv4]/ensure: created", "<13>Oct 14 13:27:37 puppet-user: Notice: /Stage[main]/Tripleo::Firewall/Tripleo::Firewall::Service_rules[tripleo_firewall]/Tripleo::Firewall::Rule[003 accept ssh from ctlplane subnet 172.16.4.0/24]/Firewall[003 accept ssh from ctlplane subnet 172.16.4.0/24 ipv4]/ensure: created", "<13>Oct 14 13:27:37 puppet-user: Notice: /Stage[main]/Tripleo::Firewall/Tripleo::Firewall::Service_rules[ovn_controller]/Tripleo::Firewall::Rule[118 neutron vxlan networks]/Firewall[118 neutron vxlan networks ipv4]/ensure: created", "<13>Oct 14 13:27:37 puppet-user: Notice: /Stage[main]/Tripleo::Firewall/Tripleo::Firewall::Service_rules[ovn_controller]/Tripleo::Firewall::Rule[118 neutron vxlan networks]/Firewall[118 neutron vxlan networks ipv6]/ensure: created", "<13>Oct 14 13:27:37 puppet-user: Notice: /Stage[main]/Tripleo::Firewall/Tripleo::Firewall::Service_rules[ovn_controller]/Tripleo::Firewall::Rule[119 neutron geneve networks]/Firewall[119 neutron geneve networks ipv4]/ensure: created", "<13>Oct 14 13:27:37 puppet-user: Notice: /Stage[main]/Tripleo::Firewall/Tripleo::Firewall::Service_rules[ovn_controller]/Tripleo::Firewall::Rule[119 neutron geneve networks]/Firewall[119 neutron geneve networks ipv6]/ensure: created", "<13>Oct 14 13:27:37 puppet-user: Notice: /Stage[main]/Tripleo::Firewall::Post/Tripleo::Firewall::Rule[998 log all]/Firewall[998 log all ipv4]/ensure: created", "<13>Oct 14 13:27:38 puppet-user: Notice: /Stage[main]/Tripleo::Firewall::Post/Tripleo::Firewall::Rule[998 log all]/Firewall[998 log all ipv6]/ensure: created", "<13>Oct 14 13:27:38 puppet-user: Notice: /Stage[main]/Tripleo::Firewall::Post/Tripleo::Firewall::Rule[999 drop all]/Firewall[999 drop all ipv4]/ensure: created", "<13>Oct 14 13:27:38 puppet-user: Notice: /Stage[main]/Tripleo::Firewall::Post/Tripleo::Firewall::Rule[999 drop all]/Firewall[999 drop all ipv6]/ensure: created", "<13>Oct 14 13:27:38 puppet-user: Notice: Applied catalog in 39.45 seconds", "<13>Oct 14 13:27:38 puppet-user: Changes:", "<13>Oct 14 13:27:38 puppet-user:             Total: 105", "<13>Oct 14 13:27:38 puppet-user: Events:", "<13>Oct 14 13:27:38 puppet-user:           Failure: 1", "<13>Oct 14 13:27:38 puppet-user:           Success: 105", "<13>Oct 14 13:27:38 puppet-user:             Total: 106", "<13>Oct 14 13:27:38 puppet-user: Resources:", "<13>Oct 14 13:27:38 puppet-user:            Failed: 1", "<13>Oct 14 13:27:38 puppet-user:         Restarted: 13", "<13>Oct 14 13:27:38 puppet-user:           Skipped: 6", "<13>Oct 14 13:27:38 puppet-user:           Changed: 92", "<13>Oct 14 13:27:38 puppet-user:       Out of sync: 92", "<13>Oct 14 13:27:38 puppet-user:             Total: 131", "<13>Oct 14 13:27:38 puppet-user: Time:", "<13>Oct 14 13:27:38 puppet-user:            Anchor: 0.00", "<13>Oct 14 13:27:38 puppet-user:          Schedule: 0.00", "<13>Oct 14 13:27:38 puppet-user:    Package manifest: 0.00", "<13>Oct 14 13:27:38 puppet-user:            Augeas: 0.02", "<13>Oct 14 13:27:38 puppet-user:           Package: 0.07", "<13>Oct 14 13:27:38 puppet-user:              Exec: 0.09", "<13>Oct 14 13:27:38 puppet-user:              File: 0.14", "<13>Oct 14 13:27:38 puppet-user:    Config retrieval: 1.25", "<13>Oct 14 13:27:38 puppet-user:          Last run: 1634218058", "<13>Oct 14 13:27:38 puppet-user:           Service: 2.06", "<13>Oct 14 13:27:38 puppet-user:    Transaction evaluation: 39.43", "<13>Oct 14 13:27:38 puppet-user:    Catalog application: 39.45", "<13>Oct 14 13:27:38 puppet-user:        Filebucket: 0.00", "<13>Oct 14 13:27:38 puppet-user:          Firewall: 5.26", "<13>Oct 14 13:27:38 puppet-user:    Concat fragment: 0.00", "<13>Oct 14 13:27:38 puppet-user:       Concat file: 0.00", "<13>Oct 14 13:27:38 puppet-user:    Certmonger certificate: 9.63", "<13>Oct 14 13:27:38 puppet-user:             Total: 39.45", "<13>Oct 14 13:27:38 puppet-user: Version:", "<13>Oct 14 13:27:38 puppet-user:            Config: 1634218017", "<13>Oct 14 13:27:38 puppet-user:            Puppet: 5.5.10"], "stdout": "", "stdout_lines": []}

2021-10-14 09:27:39.292715 | 52540053-999d-6ecf-ddc3-00000000350a |     TIMING | Wait for puppet host configuration to finish | hci-3-edge-1 | 0:12:37.343971 | 47.10s

PLAY RECAP *********************************************************************
hci-1-edge-1               : ok=237  changed=127  unreachable=0    failed=1    skipped=121  rescued=0    ignored=0   
hci-2-edge-1               : ok=227  changed=127  unreachable=0    failed=1    skipped=121  rescued=0    ignored=0   
hci-3-edge-1               : ok=227  changed=127  unreachable=0    failed=1    skipped=121  rescued=0    ignored=0   
undercloud                 : ok=238  changed=102  unreachable=0    failed=0    skipped=21   rescued=0    ignored=0   

2021-10-14 09:27:39.426985 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Summary Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2021-10-14 09:27:39.427193 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Total Tasks: 916        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2021-10-14 09:27:39.427299 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Elapsed Time: 0:12:37.478562 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2021-10-14 09:27:39.427378 |                                 UUID |       Info |       Host |   Task Name |   Run Time
2021-10-14 09:27:39.427455 | 52540053-999d-6ecf-ddc3-00000000301b |    SUMMARY | hci-3-edge-1 | Pre-fetch all the containers | 331.13s
2021-10-14 09:27:39.427532 | 52540053-999d-6ecf-ddc3-000000002fd8 |    SUMMARY | hci-2-edge-1 | Pre-fetch all the containers | 327.66s
2021-10-14 09:27:39.427604 | 52540053-999d-6ecf-ddc3-00000000301b |    SUMMARY | hci-1-edge-1 | Pre-fetch all the containers | 236.23s
2021-10-14 09:27:39.427673 | 52540053-999d-6ecf-ddc3-0000000033dc |    SUMMARY | hci-1-edge-1 | Wait for puppet host configuration to finish | 97.87s
2021-10-14 09:27:39.427742 | 52540053-999d-6ecf-ddc3-0000000034c5 |    SUMMARY | hci-2-edge-1 | Wait for puppet host configuration to finish | 47.56s
2021-10-14 09:27:39.427818 | 52540053-999d-6ecf-ddc3-00000000350a |    SUMMARY | hci-3-edge-1 | Wait for puppet host configuration to finish | 47.10s
2021-10-14 09:27:39.427950 | 52540053-999d-6ecf-ddc3-000000000051 |    SUMMARY | hci-1-edge-1 | tripleo-network-config : Run NetworkConfig script | 30.22s
2021-10-14 09:27:39.428014 | 52540053-999d-6ecf-ddc3-000000000051 |    SUMMARY | hci-3-edge-1 | tripleo-network-config : Run NetworkConfig script | 30.13s
2021-10-14 09:27:39.428096 | 52540053-999d-6ecf-ddc3-000000000051 |    SUMMARY | hci-2-edge-1 | tripleo-network-config : Run NetworkConfig script | 30.12s
2021-10-14 09:27:39.428164 | 52540053-999d-6ecf-ddc3-0000000016cc |    SUMMARY | hci-2-edge-1 | redhat-subscription : Manage Red Hat subscription | 28.29s
2021-10-14 09:27:39.428239 | 52540053-999d-6ecf-ddc3-000000001771 |    SUMMARY | hci-3-edge-1 | redhat-subscription : Manage Red Hat subscription | 26.31s
2021-10-14 09:27:39.428310 | 52540053-999d-6ecf-ddc3-000000001720 |    SUMMARY | hci-1-edge-1 | redhat-subscription : Manage Red Hat subscription | 25.91s
2021-10-14 09:27:39.428377 | 52540053-999d-6ecf-ddc3-00000000313c |    SUMMARY | hci-3-edge-1 | ipaclient : Install - Ensure that IPA client packages are installed | 11.66s
2021-10-14 09:27:39.428451 | 52540053-999d-6ecf-ddc3-0000000030c5 |    SUMMARY | hci-1-edge-1 | ipaclient : Install - Ensure that IPA client packages are installed | 11.31s
2021-10-14 09:27:39.428520 | 52540053-999d-6ecf-ddc3-00000000304d |    SUMMARY | hci-2-edge-1 | ipaclient : Install - Ensure that IPA client packages are installed | 10.97s
2021-10-14 09:27:39.428598 | 52540053-999d-6ecf-ddc3-00000000177b |    SUMMARY | hci-3-edge-1 | redhat-subscription : SATELLITE 6 | Install katello-host-tools | 10.50s
2021-10-14 09:27:39.428663 | 52540053-999d-6ecf-ddc3-00000000172a |    SUMMARY | hci-1-edge-1 | redhat-subscription : SATELLITE 6 | Install katello-host-tools | 10.33s
2021-10-14 09:27:39.428725 | 52540053-999d-6ecf-ddc3-0000000016d6 |    SUMMARY | hci-2-edge-1 | redhat-subscription : SATELLITE 6 | Install katello-host-tools | 10.02s
2021-10-14 09:27:39.428789 | 52540053-999d-6ecf-ddc3-000000003055 |    SUMMARY | hci-2-edge-1 | ipaclient : Install - Configure NTP | 7.30s
2021-10-14 09:27:39.428866 | 52540053-999d-6ecf-ddc3-0000000030cd |    SUMMARY | hci-1-edge-1 | ipaclient : Install - Configure NTP | 7.18s
2021-10-14 09:27:39.428926 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ End Summary Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2021-10-14 09:27:39.429017 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ State Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2021-10-14 09:27:39.429085 | ~~~~~~~~~~~~~~~~~~ Number of nodes which did not deploy successfully: 3 ~~~~~~~~~~~~~~~~~
2021-10-14 09:27:39.429142 |  The following node(s) had failures: hci-1-edge-1, hci-2-edge-1, hci-3-edge-1
2021-10-14 09:27:39.429203 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Host overcloud.redhat.local not found in /home/stack/.ssh/known_hosts
Ansible failed, check log at /var/lib/mistral/edge-1/ansible.log.Overcloud Endpoint: https://overcloud.redhat.local:13000
Overcloud Horizon Dashboard URL: https://overcloud.redhat.local:443/dashboard
Overcloud rc file: /home/stack/edge-1rc
Overcloud Deployed with error
Overcloud configuration failed.
END return value: 1

Comment 3 Ade Lee 2021-10-25 21:35:06 UTC

* What version of IPA server is this?
* Has the relevant ACL been added for https://pagure.io/freeipa/issue/8098
* Do the reverse DNS entries exist?
* What does the  /etc/named/ipa-options-ext.conf  look like ie. https://access.redhat.com/solutions/5753431 ?

There are a bunch of denied messages in the ipa log:

Oct 14 09:09:25 ipa-server named-pkcs11[4774]: client @0x7f5ef01a4c10 172.16.1.92#58286 (host-172-16-1-92): query (cache) 'host-172-16-1-92/A/IN' denied
Oct 14 09:09:25 ipa-server named-pkcs11[4774]: client @0x7f5ee8593ac0 172.16.1.92#58286 (host-172-16-1-92): query (cache) 'host-172-16-1-92/AAAA/IN' denied
Oct 14 09:09:26 ipa-server named-pkcs11[4774]: client @0x7f5ef01a4c10 172.16.1.91#56434 (host-172-16-1-91): query (cache) 'host-172-16-1-91/A/IN' denied
Oct 14 09:09:26 ipa-server named-pkcs11[4774]: client @0x7f5f037b06f0 172.16.1.91#56434 (host-172-16-1-91): query (cache) 'host-172-16-1-91/AAAA/IN' denied
Oct 14 09:10:48 ipa-server named-pkcs11[4774]: client @0x7f5ee8593ac0 172.16.1.93#57209 (2.rhel.pool.ntp.org): query (cache) '2.rhel.pool.ntp.org/AAAA/IN' denied
Oct 14 09:10:48 ipa-server named-pkcs11[4774]: client @0x7f5f037b06f0 172.16.1.93#57209 (2.rhel.pool.ntp.org): query (cache) '2.rhel.pool.ntp.org/A/IN' denied
Oct 14 09:10:48 ipa-server named-pkcs11[4774]: client @0x7f5f037b06f0 172.16.1.92#41176 (2.rhel.pool.ntp.org): query (cache) '2.rhel.pool.ntp.org/A/IN' denied
Oct 14 09:10:48 ipa-server named-pkcs11[4774]: client @0x7f5ee8593ac0 172.16.1.92#41176 (2.rhel.pool.ntp.org): query (cache) '2.rhel.pool.ntp.org/AAAA/IN' denied
Oct 14 09:10:51 ipa-server named-pkcs11[4774]: client @0x7f5ee8593ac0 172.16.1.91#41728 (2.rhel.pool.ntp.org): query (cache) '2.rhel.pool.ntp.org/A/IN' denied
Oct 14 09:10:51 ipa-server named-pkcs11[4774]: client @0x7f5f037b06f0 172.16.1.91#41728 (2.rhel.pool.ntp.org): query (cache) '2.rhel.pool.ntp.org/AAAA/IN' denied
Oct 14 09:13:23 ipa-server named-pkcs11[4774]: client @0x7f5f037b06f0 172.16.1.93#57657 (2.rhel.pool.ntp.org): query (cache) '2.rhel.pool.ntp.org/A/IN' denied
Oct 14 09:13:23 ipa-server named-pkcs11[4774]: client @0x7f5ef01a4c10 172.16.1.93#57657 (2.rhel.pool.ntp.org): query (cache) '2.rhel.pool.ntp.org/AAAA/IN' denied
...

which make me wonder about /etc/named/ipa-options-ext.conf

Comment 4 Sadique Puthen 2021-10-26 02:48:42 UTC

The problem in this bz is resolved by:

/etc/named/ipa-options-ext.conf

allow-recursion { any; };
allow-query-cache { any; };
allow-query { any; };

And by setting

IdMModifyDNS: True

Both need to be covered by Doc properly in the context of DCN deployment. They are not required to enable TLS-E in a non-DCN context.

Comment 5 Ade Lee 2021-10-26 15:34:04 UTC

Thanks Sadique,

I'm going to update this BZ to be a doc bug and assign it to Roger.

Comment 6 Sadique Puthen 2021-10-26 15:45:18 UTC

(In reply to Ade Lee from comment #5)
> Thanks Sadique,
> 
> I'm going to update this BZ to be a doc bug and assign it to Roger.

So any advise on if the customer sets "IdMModifyDNS: False" what needs to be populated upfront in the DNS records for deployment to succeed? Or are we going to document and recommend to set it to "true"?

Comment 7 Roger Heslop 2021-11-01 14:39:25 UTC

Hey Sadique —

Thanks for raising this question. Because we have not yet validated and documented the use-case of using an external DNS server, I'm going to go ahead and recommend within our documentation that IdMModifyDNS be set to "True".

Comment 14 Roger Heslop 2021-12-01 14:34:11 UTC

Information is published in 16.1 and 16.2 revisions of DCN documentation.