Bug 2075122 - [RHOSP 17.0] OSP can't connect to the remote QDR due to certificate permissions set incorrectly
Summary: [RHOSP 17.0] OSP can't connect to the remote QDR due to certificate permissio...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: puppet-tripleo
Version: 17.0 (Wallaby)
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: ga
: 17.0
Assignee: Martin Magr
QA Contact: Leonid Natapov
Joanne O'Flynn
URL:
Whiteboard:
Depends On:
Blocks: 1949169 2040605 2116323 2129165
TreeView+ depends on / blocked
 
Reported: 2022-04-13 16:28 UTC by Leonid Natapov
Modified: 2022-09-22 18:33 UTC (History)
10 users (show)

Fixed In Version: puppet-tripleo-14.2.3-0.20220718160752.41752a3.el9ost
Doc Type: No Doc Update
Doc Text:
Clone Of:
: 2116323 2129165 (view as bug list)
Environment:
Last Closed: 2022-09-21 12:20:43 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
OpenStack gerrit 847585 0 None MERGED Correct TLS cert permission 2022-11-30 05:08:27 UTC
Red Hat Issue Tracker OSP-14668 0 None None None 2022-04-13 16:32:34 UTC
Red Hat Product Errata RHEA-2022:6543 0 None None None 2022-09-21 12:21:12 UTC

Description Leonid Natapov 2022-04-13 16:28:10 UTC
AFter deploying OSP17 with STF,metrics_qdr log reports error in connecting to the server side.
-------------------------------------------------------------------------------------
2022-04-13 11:47:13.924624 +0000 SERVER (error) SSL CA configuration failed for connection [C1] to default-interconnect-5671-service-telemetry.apps.leonidcluster.lab.upshift.rdu2.redhat.com:443
2022-04-13 11:47:13.925460 +0000 SERVER (error) [C1] Connection aborted due to internal setup error
2022-04-13 11:47:13.925544 +0000 SERVER (info) [C1] Connection to default-interconnect-5671-service-telemetry.apps.leonidcluster.lab.upshift.rdu2.redhat.com:443 failed: amqp:connection:framing-error Expected AMQP protocol header: no protocol header found (connection aborted)
--------------------------------------------------------------------------------

This happens because certificate persmissions are set incorrectly.

[root@ctrl-0-16-2 certs]# ls -l /var/lib/config-data/puppet-generated/metrics_qdr/etc/pki/tls/certs/CA_sslProfile.pem
-rw-------. 1 root root 1326 Apr 13 15:34 /var/lib/config-data/puppet-generated/metrics_qdr/etc/pki/tls/certs/CA_sslProfile.pem

After changing permissions problem was solved:

[root@ctrl-0-16-2 certs]#  chmod -R o+rx /var/lib/config-data/puppet-generated/metrics_qdr/etc/pki/tls/

The problems seems to be here:
https://github.com/openstack/puppet-tripleo/blob/stable/wallaby/manifests/profile/base/metrics/qdr.pp#L212


Fix should be backported to OSP16.1 and OSP16.2 because once qdrouterd container will updated to newer version  this problem will also appear in 16.1 and 16.2

Comment 4 Yaniv Kaul 2022-05-11 12:40:33 UTC
(In reply to Chris Sibbitt from comment #2)
> Problem likely here:
> 
> https://github.com/openstack/puppet-tripleo/blob/stable/wallaby/manifests/
> profile/base/metrics/qdr.pp#L212
> https://github.com/openstack/puppet-tripleo/blob/stable/wallaby/manifests/
> profile/base/metrics/qdr.pp#L220
> 
> Puppet runs as root, but metrics_qdr runs as qdrouterd

Any updates? Is it a regression from 16.2?

Comment 5 Leonid Natapov 2022-05-17 12:53:19 UTC
(In reply to Yaniv Kaul from comment #4)
> (In reply to Chris Sibbitt from comment #2)
> > Problem likely here:
> > 
> > https://github.com/openstack/puppet-tripleo/blob/stable/wallaby/manifests/
> > profile/base/metrics/qdr.pp#L212
> > https://github.com/openstack/puppet-tripleo/blob/stable/wallaby/manifests/
> > profile/base/metrics/qdr.pp#L220
> > 
> > Puppet runs as root, but metrics_qdr runs as qdrouterd
> 
> Any updates? Is it a regression from 16.2?

Hey Yaniv.
Right now the issue happens in 17.0 only because we use there a different version of qdrouterd that allows us to use certificates. 
In 16.2 issue not exist now but when qdrouterd will be updated in 16.2, it will also appear in 16.2, that's why I think it should be fixed also for 16.2

Comment 6 Leif Madsen 2022-05-18 13:50:40 UTC
(In reply to Leonid Natapov from comment #5)
> (In reply to Yaniv Kaul from comment #4)
> > (In reply to Chris Sibbitt from comment #2)
> > > Problem likely here:
> > > 
> > > https://github.com/openstack/puppet-tripleo/blob/stable/wallaby/manifests/
> > > profile/base/metrics/qdr.pp#L212
> > > https://github.com/openstack/puppet-tripleo/blob/stable/wallaby/manifests/
> > > profile/base/metrics/qdr.pp#L220
> > > 
> > > Puppet runs as root, but metrics_qdr runs as qdrouterd
> > 
> > Any updates? Is it a regression from 16.2?
> 
> Hey Yaniv.
> Right now the issue happens in 17.0 only because we use there a different
> version of qdrouterd that allows us to use certificates. 
> In 16.2 issue not exist now but when qdrouterd will be updated in 16.2, it
> will also appear in 16.2, that's why I think it should be fixed also for 16.2

I agree this should be backported. I'm not sure if we're going to update the qdrouterd release in these versions if everything continues to work fine, but we'd hate to get surprised by a failure if we do end up updating them for some reason.

Comment 19 Leonid Natapov 2022-08-07 11:53:20 UTC
Fixed.

[root@controller-0 heat-admin]# ls -l /var/lib/config-data/puppet-generated/metrics_qdr/etc/pki/tls/certs/CA_sslProfile.pem
-rw-r--r--. 1 root root 1326 Aug  7 06:08 /var/lib/config-data/puppet-generated/metrics_qdr/etc/pki/tls/certs/CA_sslProfile.pem

Comment 23 errata-xmlrpc 2022-09-21 12:20:43 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Release of components for Red Hat OpenStack Platform 17.0 (Wallaby)), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2022:6543


Note You need to log in before you can comment on or make changes to this bug.