Bug 2075122

Summary:	[RHOSP 17.0] OSP can't connect to the remote QDR due to certificate permissions set incorrectly
Product:	Red Hat OpenStack	Reporter:	Leonid Natapov <lnatapov>
Component:	puppet-tripleo	Assignee:	Martin Magr <mmagr>
Status:	CLOSED ERRATA	QA Contact:	Leonid Natapov <lnatapov>
Severity:	urgent	Docs Contact:	Joanne O'Flynn <joflynn>
Priority:	urgent
Version:	17.0 (Wallaby)	CC:	astillma, csibbitt, jjoyce, jschluet, lmadsen, mmagr, pgrist, slinaber, stchen, tvignaud
Target Milestone:	ga	Keywords:	TestBlocker, Triaged
Target Release:	17.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	puppet-tripleo-14.2.3-0.20220718160752.41752a3.el9ost	Doc Type:	No Doc Update
Doc Text:		Story Points:	---
Clone Of:
Clones:	2116323 2129165 (view as bug list)		Environment:
Last Closed:	2022-09-21 12:20:43 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1949169, 2040605, 2116323, 2129165

Description Leonid Natapov 2022-04-13 16:28:10 UTC

AFter deploying OSP17 with STF,metrics_qdr log reports error in connecting to the server side.
-------------------------------------------------------------------------------------
2022-04-13 11:47:13.924624 +0000 SERVER (error) SSL CA configuration failed for connection [C1] to default-interconnect-5671-service-telemetry.apps.leonidcluster.lab.upshift.rdu2.redhat.com:443
2022-04-13 11:47:13.925460 +0000 SERVER (error) [C1] Connection aborted due to internal setup error
2022-04-13 11:47:13.925544 +0000 SERVER (info) [C1] Connection to default-interconnect-5671-service-telemetry.apps.leonidcluster.lab.upshift.rdu2.redhat.com:443 failed: amqp:connection:framing-error Expected AMQP protocol header: no protocol header found (connection aborted)
--------------------------------------------------------------------------------

This happens because certificate persmissions are set incorrectly.

[root@ctrl-0-16-2 certs]# ls -l /var/lib/config-data/puppet-generated/metrics_qdr/etc/pki/tls/certs/CA_sslProfile.pem
-rw-------. 1 root root 1326 Apr 13 15:34 /var/lib/config-data/puppet-generated/metrics_qdr/etc/pki/tls/certs/CA_sslProfile.pem

After changing permissions problem was solved:

[root@ctrl-0-16-2 certs]#  chmod -R o+rx /var/lib/config-data/puppet-generated/metrics_qdr/etc/pki/tls/

The problems seems to be here:
https://github.com/openstack/puppet-tripleo/blob/stable/wallaby/manifests/profile/base/metrics/qdr.pp#L212


Fix should be backported to OSP16.1 and OSP16.2 because once qdrouterd container will updated to newer version  this problem will also appear in 16.1 and 16.2

Comment 2 Chris Sibbitt 2022-04-13 16:34:31 UTC

Problem likely here:

https://github.com/openstack/puppet-tripleo/blob/stable/wallaby/manifests/profile/base/metrics/qdr.pp#L212
https://github.com/openstack/puppet-tripleo/blob/stable/wallaby/manifests/profile/base/metrics/qdr.pp#L220

Puppet runs as root, but metrics_qdr runs as qdrouterd

Comment 4 Yaniv Kaul 2022-05-11 12:40:33 UTC

(In reply to Chris Sibbitt from comment #2)
> Problem likely here:
> 
> https://github.com/openstack/puppet-tripleo/blob/stable/wallaby/manifests/
> profile/base/metrics/qdr.pp#L212
> https://github.com/openstack/puppet-tripleo/blob/stable/wallaby/manifests/
> profile/base/metrics/qdr.pp#L220
> 
> Puppet runs as root, but metrics_qdr runs as qdrouterd

Any updates? Is it a regression from 16.2?

Comment 5 Leonid Natapov 2022-05-17 12:53:19 UTC

(In reply to Yaniv Kaul from comment #4)
> (In reply to Chris Sibbitt from comment #2)
> > Problem likely here:
> > 
> > https://github.com/openstack/puppet-tripleo/blob/stable/wallaby/manifests/
> > profile/base/metrics/qdr.pp#L212
> > https://github.com/openstack/puppet-tripleo/blob/stable/wallaby/manifests/
> > profile/base/metrics/qdr.pp#L220
> > 
> > Puppet runs as root, but metrics_qdr runs as qdrouterd
> 
> Any updates? Is it a regression from 16.2?

Hey Yaniv.
Right now the issue happens in 17.0 only because we use there a different version of qdrouterd that allows us to use certificates. 
In 16.2 issue not exist now but when qdrouterd will be updated in 16.2, it will also appear in 16.2, that's why I think it should be fixed also for 16.2

Comment 6 Leif Madsen 2022-05-18 13:50:40 UTC

(In reply to Leonid Natapov from comment #5)
> (In reply to Yaniv Kaul from comment #4)
> > (In reply to Chris Sibbitt from comment #2)
> > > Problem likely here:
> > > 
> > > https://github.com/openstack/puppet-tripleo/blob/stable/wallaby/manifests/
> > > profile/base/metrics/qdr.pp#L212
> > > https://github.com/openstack/puppet-tripleo/blob/stable/wallaby/manifests/
> > > profile/base/metrics/qdr.pp#L220
> > > 
> > > Puppet runs as root, but metrics_qdr runs as qdrouterd
> > 
> > Any updates? Is it a regression from 16.2?
> 
> Hey Yaniv.
> Right now the issue happens in 17.0 only because we use there a different
> version of qdrouterd that allows us to use certificates. 
> In 16.2 issue not exist now but when qdrouterd will be updated in 16.2, it
> will also appear in 16.2, that's why I think it should be fixed also for 16.2

I agree this should be backported. I'm not sure if we're going to update the qdrouterd release in these versions if everything continues to work fine, but we'd hate to get surprised by a failure if we do end up updating them for some reason.

Comment 19 Leonid Natapov 2022-08-07 11:53:20 UTC

Fixed.

[root@controller-0 heat-admin]# ls -l /var/lib/config-data/puppet-generated/metrics_qdr/etc/pki/tls/certs/CA_sslProfile.pem
-rw-r--r--. 1 root root 1326 Aug  7 06:08 /var/lib/config-data/puppet-generated/metrics_qdr/etc/pki/tls/certs/CA_sslProfile.pem

Comment 23 errata-xmlrpc 2022-09-21 12:20:43 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Release of components for Red Hat OpenStack Platform 17.0 (Wallaby)), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2022:6543