Bug 1661475

Summary: [OSP14] [TLS everywhere] Overcloud installation failed on "Run puppet host configuration for step 1" - "Could not evaluate: Could not get certificate: Server at https://freeipa-0.redhat.local/ipa/xml failed request, will retry: 4001 (RPC failed at server)
Product: Red Hat OpenStack Reporter: Artem Hrechanychenko <ahrechan>
Component: SecurityAssignee: Product Security OpenStack Team <prodsec-openstack>
Status: CLOSED WORKSFORME QA Contact: nlevinki <nlevinki>
Severity: high Docs Contact:
Priority: high    
Version: 14.0 (Rocky)CC: ahrechan, hrybacki, jagee, jjoyce, josorior, jpadman, lhh, mburns, slong
Target Milestone: ---Keywords: ZStream
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-01-21 15:11:27 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
freeipa logs from ipa node, undercloud, overcloud compute node. ansible deployment logs and failures none

Description Artem Hrechanychenko 2018-12-21 10:27:58 UTC
Created attachment 1516076 [details]
freeipa logs from ipa node, undercloud, overcloud compute node. ansible deployment logs and failures

Description of problem:
OSP14
TLS everywhere scenario
3ctrl+3comp+3ceph+1freeipa node

Overcloud deployment failed


For all nodes:
 "Warning: Could not get certificate: Execution of '/usr/bin/getcert request -I libvirt-vnc-server-cert -f /etc/pki/libvirt-vnc/server-cert.pem -c IPA -N CN=compute-0.internalapi.redhat.local -K libvirt-vnc/compute-0.internalapi.redhat.local -D compute-0.internalapi.redhat.local -C \"systemctl reload libvirtd\" -w -k /etc/pki/libvirt-vnc/server-key.pem -F /etc/pki/CA/certs/vnc.crt' returned 3: New signing request \"libvirt-vnc-server-cert\" added.",
    "Error: /Stage[main]/Tripleo::Profile::Base::Certmonger_user/Tripleo::Certmonger::Libvirt_vnc[libvirt-vnc-server-cert]/Certmonger_certificate[libvirt-vnc-server-cert]: Could not evaluate: Could not get certificate: Server at https://freeipa-0.redhat.local/ipa/xml failed request, will retry: 4001 (RPC failed at server.  The host 'compute-0.internalapi.redhat.local' does not exist to add a service to.).",
...
    "Warning: Could not get certificate: Execution of '/usr/bin/getcert request -I libvirt-server-cert -f /etc/pki/libvirt/servercert.pem -c IPA -N CN=compute-1.internalapi.redhat.local -K libvirt/compute-1.internalapi.redhat.local -D compute-1.internalapi.redhat.local -C \"true\" -w -k /etc/pki/libvirt/private/serverkey.pem' returned 3: New signing request \"libvirt-server-cert\" added.",
    "Error: /Stage[main]/Tripleo::Profile::Base::Certmonger_user/Tripleo::Certmonger::Libvirt[libvirt-server-cert]/Certmonger_certificate[libvirt-server-cert]: Could not evaluate: Could not get certificate: Server at https://freeipa-0.redhat.local/ipa/xml failed request, will retry: 4001 (RPC failed at server.  The host 'compute-1.internalapi.redhat.local' does not exist to add a service to.).",
...
   "Warning: Could not get certificate: Execution of '/usr/bin/getcert request -I libvirt-vnc-server-cert -f /etc/pki/libvirt-vnc/server-cert.pem -c IPA -N CN=compute-2.internalapi.redhat.local -K libvirt-vnc/compute-2.internalapi.redhat.local -D compute-2.internalapi.redhat.local -C \"systemctl reload libvirtd\" -w -k /etc/pki/libvirt-vnc/server-key.pem -F /etc/pki/CA/certs/vnc.crt' returned 3: New signing request \"libvirt-vnc-server-cert\" added.",
    "Error: /Stage[main]/Tripleo::Profile::Base::Certmonger_user/Tripleo::Certmonger::Libvirt_vnc[libvirt-vnc-server-cert]/Certmonger_certificate[libvirt-vnc-server-cert]: Could not evaluate: Could not get certificate: Server at https://freeipa-0.redhat.local/ipa/xml failed request, will retry: 4001 (RPC failed at server.  The host 'compute-2.internalapi.redhat.local' does not exist to add a service to.).",
...
    "Warning: Could not get certificate: Execution of '/usr/bin/getcert request -I libvirt-vnc-client-cert -f /etc/pki/libvirt-vnc/client-cert.pem -c IPA -N CN=controller-0.internalapi.redhat.local -K libvirt-vnc/controller-0.internalapi.redhat.local -D controller-0.internalapi.redhat.local -C \"systemctl reload libvirtd\" -w -k /etc/pki/libvirt-vnc/client-key.pem -F /etc/pki/CA/certs/vnc.crt' returned 3: New signing request \"libvirt-vnc-client-cert\" added.",
    "Error: /Stage[main]/Tripleo::Profile::Base::Certmonger_user/Tripleo::Certmonger::Libvirt_vnc[libvirt-vnc-client-cert]/Certmonger_certificate[libvirt-vnc-client-cert]: Could not evaluate: Could not get certificate: Server at https://freeipa-0.redhat.local/ipa/xml failed request, will retry: 4001 (RPC failed at server.  The host 'controller-0.internalapi.redhat.local' does not exist to add a service to.).",
...



Version-Release number of selected component (if applicable):
RHOSP14 puddle 2018-12-17.1

How reproducible:
Always

Steps to Reproduce:
1.Prepare FreeIPA node, register it in Undercloud, deploy Undercloud
2.Deploy Overcloud with tls everywhere

Actual results:
Failed to install,The host '*host*.internalapi.redhat.local' does not exist to add a service to

Expected results:
Pass

Additional info:

Comment 2 Juan Antonio Osorio 2018-12-22 11:10:52 UTC
The relevant bit of the error message is: The host 'controller-0.internalapi.redhat.local' does not exist to add a service to.).",

Is that entry in FreeIPA?

You can check by doing: ipa host-find

with freeipa's admin credentials.

That entry should have been created by novajoin though, what version of novajoin do you have? and what version of t-h-t?

Comment 7 Juan Antonio Osorio 2019-01-11 14:51:41 UTC
were the services created in freeipa?

ipa service-find

Comment 8 Juan Antonio Osorio 2019-01-11 14:55:04 UTC
Could we get an environment that reproduces this issue? Seems some hosts are missing (the ones with internalapi.redhat.local domain).

Comment 9 Artem Hrechanychenko 2019-01-21 15:11:27 UTC
I'm going to close this bz because last runs was good in our OSP14 automation. 
will open in case of new reproduces