Bug 1562394 - OSP13 Deployment with TLS everywhere fails - Could not evaluate: The certificate * wasn't found in the list.
Summary: OSP13 Deployment with TLS everywhere fails - Could not evaluate: The certifi...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: instack-undercloud
Version: 13.0 (Queens)
Hardware: x86_64
OS: Linux
urgent
urgent
Target Milestone: beta
: 13.0 (Queens)
Assignee: Juan Antonio Osorio
QA Contact: Artem Hrechanychenko
URL:
Whiteboard:
Depends On:
Blocks: 1488826 1573583
TreeView+ depends on / blocked
 
Reported: 2018-03-30 14:11 UTC by Artem Hrechanychenko
Modified: 2019-05-13 09:54 UTC (History)
15 users (show)

Fixed In Version: instack-undercloud-8.4.1-3.el7ost, python-novajoin-1.0.18-2.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1573583 (view as bug list)
Environment:
Last Closed: 2018-06-27 13:49:35 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
Logs (16.65 MB, application/x-gzip)
2018-03-30 14:11 UTC, Artem Hrechanychenko
no flags Details
ipa logs and /var/log/messages (1.61 MB, application/x-xz)
2018-04-13 13:59 UTC, Artem Hrechanychenko
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Launchpad 1760118 0 None None None 2018-03-30 14:13:56 UTC
OpenStack gerrit 564137 0 'None' MERGED Add a retry loop when fetching vendor_data from novajoin 2020-12-03 06:00:32 UTC
OpenStack gerrit 564766 0 'None' MERGED novajoin: Add higher default timeout for nova vendordata plugins 2020-12-03 06:00:59 UTC
OpenStack gerrit 565130 0 'None' MERGED novajoin: Add higher default timeout for nova vendordata plugins 2020-12-03 06:00:33 UTC
Red Hat Product Errata RHEA-2018:2086 0 None None None 2018-06-27 13:50:27 UTC

Description Artem Hrechanychenko 2018-03-30 14:11:44 UTC
Created attachment 1415110 [details]
Logs

Description of problem:
Cannot Deploy OSP13 with TLS everywhere

(undercloud) [stack@undercloud-0 ~]$ cat file |grep "Could not evaluate: The certificate"
            "Error: /Stage[main]/Tripleo::Certmonger::Mysql/Certmonger_certificate[mysql]: Could not evaluate: The certificate 'mysql' wasn't found in the list.", 
            "Error: /Stage[main]/Tripleo::Certmonger::Rabbitmq/Certmonger_certificate[rabbitmq]: Could not evaluate: The certificate 'rabbitmq' wasn't found in the list.", 
            "Error: /Stage[main]/Tripleo::Certmonger::Redis/Certmonger_certificate[redis]: Could not evaluate: The certificate 'redis' wasn't found in the list.", 
            "Error: /Stage[main]/Tripleo::Certmonger::Neutron/Certmonger_certificate[neutron]: Could not evaluate: The certificate 'neutron' wasn't found in the list.", 
            "Error: /Stage[main]/Tripleo::Profile::Base::Certmonger_user/Tripleo::Certmonger::Httpd[httpd-ctlplane]/Certmonger_certificate[httpd-ctlplane]: Could not evaluate: The certificate 'httpd-ctlplane' wasn't found in the list.", 
            "Error: /Stage[main]/Tripleo::Profile::Base::Certmonger_user/Tripleo::Certmonger::Httpd[httpd-external]/Certmonger_certificate[httpd-external]: Could not evaluate: The certificate 'httpd-external' wasn't found in the list.", 
            "Error: /Stage[main]/Tripleo::Profile::Base::Certmonger_user/Tripleo::Certmonger::Httpd[httpd-internal_api]/Certmonger_certificate[httpd-internal_api]: Could not evaluate: The certificate 'httpd-internal_api' wasn't found in the list.", 
            "Error: /Stage[main]/Tripleo::Profile::Base::Certmonger_user/Tripleo::Certmonger::Httpd[httpd-management]/Certmonger_certificate[httpd-management]: Could not evaluate: The certificate 'httpd-management' wasn't found in the list.", 
            "Error: /Stage[main]/Tripleo::Profile::Base::Certmonger_user/Tripleo::Certmonger::Httpd[httpd-storage]/Certmonger_certificate[httpd-storage]: Could not evaluate: The certificate 'httpd-storage' wasn't found in the list.", 
            "Error: /Stage[main]/Tripleo::Profile::Base::Certmonger_user/Tripleo::Certmonger::Httpd[httpd-storage_mgmt]/Certmonger_certificate[httpd-storage_mgmt]: Could not evaluate: The certificate 'httpd-storage_mgmt' wasn't found in the list.", 
            "Error: /Stage[main]/Tripleo::Profile::Base::Certmonger_user/Tripleo::Certmonger::Libvirt_vnc[libvirt-vnc-client-cert]/Certmonger_certificate[libvirt-vnc-client-cert]: Could not evaluate: The certificate 'libvirt-vnc-client-cert' wasn't found in the list.", 
            "Error: /Stage[main]/Tripleo::Profile::Base::Certmonger_user/Tripleo::Certmonger::Haproxy[haproxy-ctlplane]/Certmonger_certificate[haproxy-ctlplane-cert]: Could not evaluate: The certificate 'haproxy-ctlplane-cert' wasn't found in the list.", 
            "Error: /Stage[main]/Tripleo::Profile::Base::Certmonger_user/Tripleo::Certmonger::Haproxy[haproxy-external]/Certmonger_certificate[haproxy-external-cert]: Could not evaluate: The certificate 'haproxy-external-cert' wasn't found in the list.", 
            "Error: /Stage[main]/Tripleo::Profile::Base::Certmonger_user/Tripleo::Certmonger::Haproxy[haproxy-internal_api]/Certmonger_certificate[haproxy-internal_api-cert]: Could not evaluate: The certificate 'haproxy-internal_api-cert' wasn't found in the list.", 
            "Error: /Stage[main]/Tripleo::Profile::Base::Certmonger_user/Tripleo::Certmonger::Haproxy[haproxy-storage]/Certmonger_certificate[haproxy-storage-cert]: Could not evaluate: The certificate 'haproxy-storage-cert' wasn't found in the list.", 
            "Error: /Stage[main]/Tripleo::Profile::Base::Certmonger_user/Tripleo::Certmonger::Haproxy[haproxy-storage_mgmt]/Certmonger_certificate[haproxy-storage_mgmt-cert]: Could not evaluate: The certificate 'haproxy-storage_mgmt-cert' wasn't found in the list.", 

Version-Release number of selected component (if applicable):
(undercloud) [stack@undercloud-0 ~]$ sudo rpm -qa "*openstack*"
openstack-nova-scheduler-17.0.2-0.20180323024604.0390d5f.el7ost.noarch
puppet-openstacklib-12.3.1-0.20180221063157.8ced16a.el7ost.noarch
openstack-glance-16.0.1-0.20180321165819.2221868.el7ost.noarch
openstack-nova-compute-17.0.2-0.20180323024604.0390d5f.el7ost.noarch
openstack-neutron-12.0.1-0.20180327195360.68b8980.el7ost.noarch
openstack-heat-engine-10.0.1-0.20180314232329.c2a66b1.el7ost.noarch
openstack-tempest-18.0.0-2.el7ost.noarch
openstack-swift-object-2.17.1-0.20180314165245.caeeb54.el7ost.noarch
openstack-keystone-13.0.1-0.20180322035847.c1d81ef.el7ost.noarch
openstack-selinux-0.8.14-1.el7ost.noarch
openstack-neutron-common-12.0.1-0.20180327195360.68b8980.el7ost.noarch
python2-openstacksdk-0.11.3-1.el7ost.noarch
openstack-heat-common-10.0.1-0.20180314232329.c2a66b1.el7ost.noarch
openstack-ironic-conductor-10.1.2-0.20180326121311.ef08927.el7ost.noarch
openstack-tripleo-image-elements-8.0.0-2.el7ost.noarch
openstack-mistral-common-6.0.1-0.20180319140929.eb59183.el7ost.noarch
openstack-tripleo-ui-8.3.1-2.el7ost.noarch
puppet-openstack_extras-12.3.1-0.20180221064243.0b9edf4.el7ost.noarch
openstack-nova-placement-api-17.0.2-0.20180323024604.0390d5f.el7ost.noarch
openstack-tripleo-puppet-elements-8.0.0-1.el7ost.noarch
openstack-tripleo-common-containers-8.5.1-0.20180326153322.91f52e9.el7ost.noarch
python-openstackclient-lang-3.14.0-1.el7ost.noarch
openstack-tripleo-common-8.5.1-0.20180326153322.91f52e9.el7ost.noarch
openstack-mistral-executor-6.0.1-0.20180319140929.eb59183.el7ost.noarch
openstack-zaqar-6.0.1-0.20180302005413.4659f9b.el7ost.noarch
openstack-nova-common-17.0.2-0.20180323024604.0390d5f.el7ost.noarch
openstack-swift-account-2.17.1-0.20180314165245.caeeb54.el7ost.noarch
openstack-neutron-ml2-12.0.1-0.20180327195360.68b8980.el7ost.noarch
openstack-heat-api-cfn-10.0.1-0.20180314232329.c2a66b1.el7ost.noarch
openstack-ironic-common-10.1.2-0.20180326121311.ef08927.el7ost.noarch
openstack-ironic-staging-drivers-0.9.0-2.el7ost.noarch
openstack-mistral-engine-6.0.1-0.20180319140929.eb59183.el7ost.noarch
openstack-ironic-api-10.1.2-0.20180326121311.ef08927.el7ost.noarch
openstack-ironic-inspector-7.2.1-0.20180302142656.397a98a.el7ost.noarch
openstack-tripleo-validations-8.4.0-1.el7ost.noarch
openstack-nova-api-17.0.2-0.20180323024604.0390d5f.el7ost.noarch
openstack-nova-conductor-17.0.2-0.20180323024604.0390d5f.el7ost.noarch
openstack-swift-proxy-2.17.1-0.20180314165245.caeeb54.el7ost.noarch
openstack-swift-container-2.17.1-0.20180314165245.caeeb54.el7ost.noarch
openstack-neutron-openvswitch-12.0.1-0.20180327195360.68b8980.el7ost.noarch
openstack-heat-api-10.0.1-0.20180314232329.c2a66b1.el7ost.noarch
python2-openstackclient-3.14.0-1.el7ost.noarch
openstack-tripleo-heat-templates-8.0.2-0.20180327213843.f25e2d8.el7ost.noarch
openstack-mistral-api-6.0.1-0.20180319140929.eb59183.el7ost.noarch


How reproducible:
Always

Steps to Reproduce:
https://rhos-ci-staging-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/DF%20Current%20release/job/DFG-df-13-deployment-7.5-virthost-3cont_3comp_3ceph-yes_UC_SSL-yes_OC_SSL-ceph-ipv4-vxlan-TLS_everywhere-RHELOSP-31826/14/

Actual results:
Deployment was fail

Expected results:
Deployment was pass

Additional info:

Comment 3 Itzik Brown 2018-04-12 11:21:31 UTC
I have the same problem - It affects also the OpenDaylight SSL/TLS feature

Comment 4 Tim Rozet 2018-04-12 19:39:19 UTC
This usually indicates the overcloud failed to enroll with freeipa or your CA (that's why it cannot get a certificate).  Can you please ensure your undercloud is enrolled, and if so, then attach the /var/log/messages and /var/log/cloud-init logs.  We need to see why the overcloud was unable to join if that is indeed the issue.

Comment 5 Artem Hrechanychenko 2018-04-13 13:59:00 UTC
Created attachment 1421400 [details]
ipa logs and /var/log/messages

Comment 6 Juan Antonio Osorio 2018-04-16 16:21:39 UTC
Could you provide an environment where this issue is presented so I can check it out?

Comment 8 Tim Rozet 2018-04-17 20:32:25 UTC
Itzik hit the same problem.  The problem is the overcloud does not join IPA. 
 The solution was to increase the timeouts for the vendor data settings.  In nova.conf:

vendordata_dynamic_connect_timeout = 30
vendordata_dynamic_read_timeout = 30

Then restart the openstack-nova-api, novajoin-notify, and novajoin-server processes.  Let me know if that fixes it.

Comment 9 Artem Hrechanychenko 2018-04-19 13:59:53 UTC
@trozet,


vendordata_dynamic_connect_timeout = 30
vendordata_dynamic_read_timeout = 30

in /etc/nova/nova.conf on undercloud node and reboot freeipa and UC node works as a w/a for me

Comment 11 Harry Rybacki 2018-04-19 21:31:41 UTC
@Nir -- we have eyes on this.

Resetting NEEDINFO from comment#6 -- Artem are you able to provide Ozz with an env. for debugging?

Comment 12 Artem Hrechanychenko 2018-04-20 13:28:52 UTC
env details were send via email

Comment 15 Harry Rybacki 2018-05-01 18:02:00 UTC
Downstream patches have merged, builds created and noted in `fixed-in`. 

Moving but to MODIFIED.

Comment 24 errata-xmlrpc 2018-06-27 13:49:35 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:2086


Note You need to log in before you can comment on or make changes to this bug.