Bug 1625732

Summary: Unable to deploy TLS OC using a containerized UC
Product: Red Hat OpenStack Reporter: Juan Antonio Osorio <josorior>
Component: openstack-tripleo-heat-templatesAssignee: Martin Schuppert <mschuppe>
Status: CLOSED ERRATA QA Contact: Archit Modi <amodi>
Severity: high Docs Contact:
Priority: high    
Version: 14.0 (Rocky)CC: bcafarel, beagles, berrange, bfournie, chjones, dasmith, dbecker, eglynn, hjensas, hrybacki, jhakimra, kchamart, lyarwood, mburns, michele, morazi, mschuppe, owalsh, rmascena, sbauza, sgordon, vromanso
Target Milestone: rcKeywords: Triaged
Target Release: 14.0 (Rocky)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-tripleo-heat-templates-9.0.0-0.20180919080946.0rc1.0rc1.el7ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-01-11 11:51:51 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1645136, 1652287    
Bug Blocks: 1517811, 1608744, 1609025    

Description Juan Antonio Osorio 2018-09-05 17:15:08 UTC
Description of problem:

For the "TLS everywhere" setup, it is required that the overcloud nodes fetch their passwords to enroll to the CA via metadata. This is no longer working for OSP14.

[root@overcloud-controller-0 log]# cat setup-ipa-client.log 
++ timeout 300 /bin/bash -c 'data=""; while [ -z "$data" ]; do sleep $[ ( $RANDOM % 10 )  + 1 ]s; data=`curl -s http://169.254.169.254/openstack/2016-10-06/vendor_data2.json 2>/dev/null`; done; echo $data'
+ data=
+ [[ 124 != 0 ]]
+ echo 'Unable to retrieve metadata'
Unable to retrieve metadata
+ exit 1
[root@overcloud-controller-0 log]# data=`curl -s http://169.254.169.254/openstack/2016-10-06/vendor_data2.json`
[root@overcloud-controller-0 log]# echo $data

[root@overcloud-controller-0 log]# 


The call to vendor_data2.json fetches the output from the vendordata plugin, which we have configured to be novajoin.

On the other hand, doing "docker logs novajoin_server" does show that novajoin is called by nova, and it creates the relevant hosts on FreeIPA. So nova DOES call the vendordata plugin.

So I think that the issue is that somewhere, the call to the novametadata gets lost.

The route seems to be there:

[root@overcloud-controller-0 heat-admin]# route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         gateway         0.0.0.0         UG    0      0        0 vlan10
10.0.0.0        0.0.0.0         255.255.255.0   U     0      0        0 vlan10
169.254.169.254 192.168.24.1    255.255.255.255 UGH   0      0        0 br-ex
172.16.0.0      0.0.0.0         255.255.255.0   U     0      0        0 vlan50
172.16.1.0      0.0.0.0         255.255.255.0   U     0      0        0 vlan30
172.16.2.0      0.0.0.0         255.255.255.0   U     0      0        0 vlan20
172.16.3.0      0.0.0.0         255.255.255.0   U     0      0        0 vlan40
172.31.0.0      0.0.0.0         255.255.255.0   U     0      0        0 docker0
192.168.24.0    0.0.0.0         255.255.255.0   U     0      0        0 br-ex



Version-Release number of selected component (if applicable):
OSP14

How reproducible:
100%

Steps to Reproduce:
1. Deploy TLS everywhere
2. try to fetch nova metadata (vendordata)

Actual results:
no response

Expected results:
should get the OTP for the node and continue the enrollment

Comment 1 Juan Antonio Osorio 2018-09-05 19:10:31 UTC
This is an issue with nova metadata not being passed.

Comment 2 Raildo Mascena de Sousa Filho 2018-09-05 19:12:32 UTC
Moving to DFG:Compute, since they take care of the nova metadata.

Comment 4 Brent Eagles 2018-09-27 13:12:54 UTC
This is a regression incurred during the switch from instack-undercloud to the containerized undercloud. Previously a rule was added to iptables to redirect traffic from port 80 on the link local interface to port 8775.

Comment 6 Martin Schuppert 2018-10-01 08:23:34 UTC
(In reply to Brent Eagles from comment #4)
> This is a regression incurred during the switch from instack-undercloud to
> the containerized undercloud. Previously a rule was added to iptables to
> redirect traffic from port 80 on the link local interface to port 8775.

In addition to the redirect we also need to config nova metadata to not have neutron metadata agent configured. 

service_metadata_proxy=false

Comment 14 Joe H. Rahme 2018-11-02 08:20:32 UTC
TLS-everywhere deployment is failing with the following error:

2018-10-30 12:41:52.247 | fatal: [undercloud-0 -> 172.16.0.37]: FAILED! => {
2018-10-30 12:41:52.249 |     "changed": true,
2018-10-30 12:41:52.252 |     "cmd": "/usr/libexec/novajoin-ipa-setup --principal admin --password 12345678 --server freeipa-0.redhat.local --realm REDHAT.LOCAL --domain redhat.local --hostname undercloud-0.redhat.local --precreate",
2018-10-30 12:41:52.255 |     "delta": "0:00:00.272060",
2018-10-30 12:41:52.258 |     "end": "2018-10-30 08:41:52.061568",
2018-10-30 12:41:52.261 |     "invocation": {
2018-10-30 12:41:52.264 |         "module_args": {
2018-10-30 12:41:52.267 |             "_raw_params": "/usr/libexec/novajoin-ipa-setup --principal admin --password 12345678 --server freeipa-0.redhat.local --realm REDHAT.LOCAL --domain redhat.local --hostname undercloud-0.redhat.local --precreate",
2018-10-30 12:41:52.269 |             "_uses_shell": true,
2018-10-30 12:41:52.272 |             "chdir": null,
2018-10-30 12:41:52.275 |             "creates": null,
2018-10-30 12:41:52.278 |             "executable": null,
2018-10-30 12:41:52.281 |             "removes": null,
2018-10-30 12:41:52.284 |             "stdin": null,
2018-10-30 12:41:52.287 |             "warn": true
2018-10-30 12:41:52.289 |         }
2018-10-30 12:41:52.292 |     },
2018-10-30 12:41:52.295 |     "rc": 1,
2018-10-30 12:41:52.298 |     "start": "2018-10-30 08:41:51.789508"
2018-10-30 12:41:52.301 | }
2018-10-30 12:41:52.304 |
2018-10-30 12:41:52.306 | STDERR:
2018-10-30 12:41:52.309 |
2018-10-30 12:41:52.312 | Traceback (most recent call last):
2018-10-30 12:41:52.315 |   File "/usr/libexec/novajoin-ipa-setup", line 70, in <module>
2018-10-30 12:41:52.318 |     console = log_mgr.get_handler('console')
2018-10-30 12:41:52.321 | AttributeError: 'module' object has no attribute 'get_handler'
2018-10-30 12:41:52.323 |
2018-10-30 12:41:52.326 |
2018-10-30 12:41:52.329 | MSG:
2018-10-30 12:41:52.332 |
2018-10-30 12:41:52.335 | non-zero return code

Comment 15 Juan Antonio Osorio 2018-11-02 08:24:00 UTC
John, that issue came up in RHEL 7.6 [1] we're working on it.

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1644747

Comment 21 errata-xmlrpc 2019-01-11 11:51:51 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:0045