Bug 1562394

Summary: OSP13 Deployment with TLS everywhere fails - Could not evaluate: The certificate * wasn't found in the list.
Product: Red Hat OpenStack Reporter: Artem Hrechanychenko <ahrechan>
Component: instack-undercloudAssignee: Juan Antonio Osorio <josorior>
Status: CLOSED ERRATA QA Contact: Artem Hrechanychenko <ahrechan>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 13.0 (Queens)CC: ahrechan, alee, hrybacki, itbrown, josorior, jschluet, kbasil, knylande, mburns, nkinder, nyechiel, pkesavar, rhel-osp-director-maint, rmascena, vitorepoma
Target Milestone: betaKeywords: Triaged
Target Release: 13.0 (Queens)   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: instack-undercloud-8.4.1-3.el7ost, python-novajoin-1.0.18-2.el7ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1573583 (view as bug list) Environment:
Last Closed: 2018-06-27 13:49:35 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1488826, 1573583    
Attachments:
Description Flags
Logs
none
ipa logs and /var/log/messages none

Description Artem Hrechanychenko 2018-03-30 14:11:44 UTC
Created attachment 1415110 [details]
Logs

Description of problem:
Cannot Deploy OSP13 with TLS everywhere

(undercloud) [stack@undercloud-0 ~]$ cat file |grep "Could not evaluate: The certificate"
            "Error: /Stage[main]/Tripleo::Certmonger::Mysql/Certmonger_certificate[mysql]: Could not evaluate: The certificate 'mysql' wasn't found in the list.", 
            "Error: /Stage[main]/Tripleo::Certmonger::Rabbitmq/Certmonger_certificate[rabbitmq]: Could not evaluate: The certificate 'rabbitmq' wasn't found in the list.", 
            "Error: /Stage[main]/Tripleo::Certmonger::Redis/Certmonger_certificate[redis]: Could not evaluate: The certificate 'redis' wasn't found in the list.", 
            "Error: /Stage[main]/Tripleo::Certmonger::Neutron/Certmonger_certificate[neutron]: Could not evaluate: The certificate 'neutron' wasn't found in the list.", 
            "Error: /Stage[main]/Tripleo::Profile::Base::Certmonger_user/Tripleo::Certmonger::Httpd[httpd-ctlplane]/Certmonger_certificate[httpd-ctlplane]: Could not evaluate: The certificate 'httpd-ctlplane' wasn't found in the list.", 
            "Error: /Stage[main]/Tripleo::Profile::Base::Certmonger_user/Tripleo::Certmonger::Httpd[httpd-external]/Certmonger_certificate[httpd-external]: Could not evaluate: The certificate 'httpd-external' wasn't found in the list.", 
            "Error: /Stage[main]/Tripleo::Profile::Base::Certmonger_user/Tripleo::Certmonger::Httpd[httpd-internal_api]/Certmonger_certificate[httpd-internal_api]: Could not evaluate: The certificate 'httpd-internal_api' wasn't found in the list.", 
            "Error: /Stage[main]/Tripleo::Profile::Base::Certmonger_user/Tripleo::Certmonger::Httpd[httpd-management]/Certmonger_certificate[httpd-management]: Could not evaluate: The certificate 'httpd-management' wasn't found in the list.", 
            "Error: /Stage[main]/Tripleo::Profile::Base::Certmonger_user/Tripleo::Certmonger::Httpd[httpd-storage]/Certmonger_certificate[httpd-storage]: Could not evaluate: The certificate 'httpd-storage' wasn't found in the list.", 
            "Error: /Stage[main]/Tripleo::Profile::Base::Certmonger_user/Tripleo::Certmonger::Httpd[httpd-storage_mgmt]/Certmonger_certificate[httpd-storage_mgmt]: Could not evaluate: The certificate 'httpd-storage_mgmt' wasn't found in the list.", 
            "Error: /Stage[main]/Tripleo::Profile::Base::Certmonger_user/Tripleo::Certmonger::Libvirt_vnc[libvirt-vnc-client-cert]/Certmonger_certificate[libvirt-vnc-client-cert]: Could not evaluate: The certificate 'libvirt-vnc-client-cert' wasn't found in the list.", 
            "Error: /Stage[main]/Tripleo::Profile::Base::Certmonger_user/Tripleo::Certmonger::Haproxy[haproxy-ctlplane]/Certmonger_certificate[haproxy-ctlplane-cert]: Could not evaluate: The certificate 'haproxy-ctlplane-cert' wasn't found in the list.", 
            "Error: /Stage[main]/Tripleo::Profile::Base::Certmonger_user/Tripleo::Certmonger::Haproxy[haproxy-external]/Certmonger_certificate[haproxy-external-cert]: Could not evaluate: The certificate 'haproxy-external-cert' wasn't found in the list.", 
            "Error: /Stage[main]/Tripleo::Profile::Base::Certmonger_user/Tripleo::Certmonger::Haproxy[haproxy-internal_api]/Certmonger_certificate[haproxy-internal_api-cert]: Could not evaluate: The certificate 'haproxy-internal_api-cert' wasn't found in the list.", 
            "Error: /Stage[main]/Tripleo::Profile::Base::Certmonger_user/Tripleo::Certmonger::Haproxy[haproxy-storage]/Certmonger_certificate[haproxy-storage-cert]: Could not evaluate: The certificate 'haproxy-storage-cert' wasn't found in the list.", 
            "Error: /Stage[main]/Tripleo::Profile::Base::Certmonger_user/Tripleo::Certmonger::Haproxy[haproxy-storage_mgmt]/Certmonger_certificate[haproxy-storage_mgmt-cert]: Could not evaluate: The certificate 'haproxy-storage_mgmt-cert' wasn't found in the list.", 

Version-Release number of selected component (if applicable):
(undercloud) [stack@undercloud-0 ~]$ sudo rpm -qa "*openstack*"
openstack-nova-scheduler-17.0.2-0.20180323024604.0390d5f.el7ost.noarch
puppet-openstacklib-12.3.1-0.20180221063157.8ced16a.el7ost.noarch
openstack-glance-16.0.1-0.20180321165819.2221868.el7ost.noarch
openstack-nova-compute-17.0.2-0.20180323024604.0390d5f.el7ost.noarch
openstack-neutron-12.0.1-0.20180327195360.68b8980.el7ost.noarch
openstack-heat-engine-10.0.1-0.20180314232329.c2a66b1.el7ost.noarch
openstack-tempest-18.0.0-2.el7ost.noarch
openstack-swift-object-2.17.1-0.20180314165245.caeeb54.el7ost.noarch
openstack-keystone-13.0.1-0.20180322035847.c1d81ef.el7ost.noarch
openstack-selinux-0.8.14-1.el7ost.noarch
openstack-neutron-common-12.0.1-0.20180327195360.68b8980.el7ost.noarch
python2-openstacksdk-0.11.3-1.el7ost.noarch
openstack-heat-common-10.0.1-0.20180314232329.c2a66b1.el7ost.noarch
openstack-ironic-conductor-10.1.2-0.20180326121311.ef08927.el7ost.noarch
openstack-tripleo-image-elements-8.0.0-2.el7ost.noarch
openstack-mistral-common-6.0.1-0.20180319140929.eb59183.el7ost.noarch
openstack-tripleo-ui-8.3.1-2.el7ost.noarch
puppet-openstack_extras-12.3.1-0.20180221064243.0b9edf4.el7ost.noarch
openstack-nova-placement-api-17.0.2-0.20180323024604.0390d5f.el7ost.noarch
openstack-tripleo-puppet-elements-8.0.0-1.el7ost.noarch
openstack-tripleo-common-containers-8.5.1-0.20180326153322.91f52e9.el7ost.noarch
python-openstackclient-lang-3.14.0-1.el7ost.noarch
openstack-tripleo-common-8.5.1-0.20180326153322.91f52e9.el7ost.noarch
openstack-mistral-executor-6.0.1-0.20180319140929.eb59183.el7ost.noarch
openstack-zaqar-6.0.1-0.20180302005413.4659f9b.el7ost.noarch
openstack-nova-common-17.0.2-0.20180323024604.0390d5f.el7ost.noarch
openstack-swift-account-2.17.1-0.20180314165245.caeeb54.el7ost.noarch
openstack-neutron-ml2-12.0.1-0.20180327195360.68b8980.el7ost.noarch
openstack-heat-api-cfn-10.0.1-0.20180314232329.c2a66b1.el7ost.noarch
openstack-ironic-common-10.1.2-0.20180326121311.ef08927.el7ost.noarch
openstack-ironic-staging-drivers-0.9.0-2.el7ost.noarch
openstack-mistral-engine-6.0.1-0.20180319140929.eb59183.el7ost.noarch
openstack-ironic-api-10.1.2-0.20180326121311.ef08927.el7ost.noarch
openstack-ironic-inspector-7.2.1-0.20180302142656.397a98a.el7ost.noarch
openstack-tripleo-validations-8.4.0-1.el7ost.noarch
openstack-nova-api-17.0.2-0.20180323024604.0390d5f.el7ost.noarch
openstack-nova-conductor-17.0.2-0.20180323024604.0390d5f.el7ost.noarch
openstack-swift-proxy-2.17.1-0.20180314165245.caeeb54.el7ost.noarch
openstack-swift-container-2.17.1-0.20180314165245.caeeb54.el7ost.noarch
openstack-neutron-openvswitch-12.0.1-0.20180327195360.68b8980.el7ost.noarch
openstack-heat-api-10.0.1-0.20180314232329.c2a66b1.el7ost.noarch
python2-openstackclient-3.14.0-1.el7ost.noarch
openstack-tripleo-heat-templates-8.0.2-0.20180327213843.f25e2d8.el7ost.noarch
openstack-mistral-api-6.0.1-0.20180319140929.eb59183.el7ost.noarch


How reproducible:
Always

Steps to Reproduce:
https://rhos-ci-staging-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/DF%20Current%20release/job/DFG-df-13-deployment-7.5-virthost-3cont_3comp_3ceph-yes_UC_SSL-yes_OC_SSL-ceph-ipv4-vxlan-TLS_everywhere-RHELOSP-31826/14/

Actual results:
Deployment was fail

Expected results:
Deployment was pass

Additional info:

Comment 3 Itzik Brown 2018-04-12 11:21:31 UTC
I have the same problem - It affects also the OpenDaylight SSL/TLS feature

Comment 4 Tim Rozet 2018-04-12 19:39:19 UTC
This usually indicates the overcloud failed to enroll with freeipa or your CA (that's why it cannot get a certificate).  Can you please ensure your undercloud is enrolled, and if so, then attach the /var/log/messages and /var/log/cloud-init logs.  We need to see why the overcloud was unable to join if that is indeed the issue.

Comment 5 Artem Hrechanychenko 2018-04-13 13:59:00 UTC
Created attachment 1421400 [details]
ipa logs and /var/log/messages

Comment 6 Juan Antonio Osorio 2018-04-16 16:21:39 UTC
Could you provide an environment where this issue is presented so I can check it out?

Comment 8 Tim Rozet 2018-04-17 20:32:25 UTC
Itzik hit the same problem.  The problem is the overcloud does not join IPA. 
 The solution was to increase the timeouts for the vendor data settings.  In nova.conf:

vendordata_dynamic_connect_timeout = 30
vendordata_dynamic_read_timeout = 30

Then restart the openstack-nova-api, novajoin-notify, and novajoin-server processes.  Let me know if that fixes it.

Comment 9 Artem Hrechanychenko 2018-04-19 13:59:53 UTC
@trozet,


vendordata_dynamic_connect_timeout = 30
vendordata_dynamic_read_timeout = 30

in /etc/nova/nova.conf on undercloud node and reboot freeipa and UC node works as a w/a for me

Comment 11 Harry Rybacki 2018-04-19 21:31:41 UTC
@Nir -- we have eyes on this.

Resetting NEEDINFO from comment#6 -- Artem are you able to provide Ozz with an env. for debugging?

Comment 12 Artem Hrechanychenko 2018-04-20 13:28:52 UTC
env details were send via email

Comment 15 Harry Rybacki 2018-05-01 18:02:00 UTC
Downstream patches have merged, builds created and noted in `fixed-in`. 

Moving but to MODIFIED.

Comment 24 errata-xmlrpc 2018-06-27 13:49:35 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:2086