Bug 1849633

Summary: Bootstrap: EXITING: [/opt/puppetlabs/puppet/bin/puppet agent --test --noop --tags no_such_tag --waitforcert 10] failed to execute properly.
Product: Red Hat Satellite Reporter: Lukáš Hellebrandt <lhellebr>
Component: PuppetAssignee: satellite6-bugs <satellite6-bugs>
Status: CLOSED WONTFIX QA Contact: Vladimír Sedmík <vsedmik>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 6.8.0CC: egolov, ekohlvan, tbrisker, zhunting
Target Milestone: UnspecifiedKeywords: Regression, Triaged
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-03-18 19:44:22 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
bootstrap.log none

Description Lukáš Hellebrandt 2020-06-22 12:10:31 UTC
Created attachment 1698286 [details]
bootstrap.log

Description of problem:
When running a bootstrap script against Sat 6.8, it fails with the error shown in the attached file. After that, the host (and a content host) is shown in the Satellite anyway.

Version-Release number of selected component (if applicable):
Sat 6.8 snap 5.0

How reproducible:
Deterministic

Steps to Reproduce:
1. Have a Satellite and a RHEL host
2. On Satellite, have necessary entities to run the bootstrap
3. # ./bootstrap.py --login=<login> \
--server <FQDN> \
--location="Default Location" \
--organization="Default Organization" \
--hostgroup="testgrp" \
--activationkey=testak \
--password=<password> \
--force \
--add-domain

Actual results:
EXITING: [/opt/puppetlabs/puppet/bin/puppet agent --test --noop --tags no_such_tag --waitforcert 10] failed to execute properly.

Expected results:
Bootstrap successful.

Comment 1 Evgeni Golov 2020-06-23 09:12:15 UTC
This does not look like a bootstrap bug, more like some issue with the Puppet and its CA.

Comment 2 Lukáš Hellebrandt 2020-06-23 09:30:14 UTC
I agree. Brad, I think this should be switched back to Puppet component.

Comment 3 Tomer Brisker 2020-07-01 13:22:42 UTC
Looks like the failure is due to self-signed cert in the chain. Ewoud, could this be related to puppet 6 adding an intermediary cert?

Info: Caching certificate for <FQDN_HOST>
Warning: Unable to fetch my node definition, but the agent run will continue:
Warning: SSL_connect returned=1 errno=0 state=error: certificate verify failed: [self signed certificate in certificate chain for /CN=Puppet Root CA: 3b5ff26c750018]
Info: Retrieving pluginfacts
Error: /File[/opt/puppetlabs/puppet/cache/facts.d]: Failed to generate additional resources using 'eval_generate': SSL_connect returned=1 errno=0 state=error: certificate verify failed: [self signed certificate in certificate chain for /CN=Puppet Root CA: 3b5ff26c750018]
Error: /File[/opt/puppetlabs/puppet/cache/facts.d]: Could not evaluate: Could not retrieve file metadata for puppet:///pluginfacts: SSL_connect returned=1 errno=0 state=error: certificate verify failed: [self signed certificate in certificate chain for /CN=Puppet Root CA: 3b5ff26c750018]
Info: Retrieving plugin
Error: /File[/opt/puppetlabs/puppet/cache/lib]: Failed to generate additional resources using 'eval_generate': SSL_connect returned=1 errno=0 state=error: certificate verify failed: [self signed certificate in certificate chain for /CN=Puppet Root CA: 3b5ff26c750018]
Error: /File[/opt/puppetlabs/puppet/cache/lib]: Could not evaluate: Could not retrieve file metadata for puppet:///plugins: SSL_connect returned=1 errno=0 state=error: certificate verify failed: [self signed certificate in certificate chain for /CN=Puppet Root CA: 3b5ff26c750018]
Info: Retrieving locales
Error: /File[/opt/puppetlabs/puppet/cache/locales]: Failed to generate additional resources using 'eval_generate': SSL_connect returned=1 errno=0 state=error: certificate verify failed: [self signed certificate in certificate chain for /CN=Puppet Root CA: 3b5ff26c750018]
Error: /File[/opt/puppetlabs/puppet/cache/locales]: Could not evaluate: Could not retrieve file metadata for puppet:///locales: SSL_connect returned=1 errno=0 state=error: certificate verify failed: [self signed certificate in certificate chain for /CN=Puppet Root CA: 3b5ff26c750018]
Error: Could not retrieve catalog from remote server: SSL_connect returned=1 errno=0 state=error: certificate verify failed: [self signed certificate in certificate chain for /CN=Puppet Root CA: 3b5ff26c750018]
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run
Error: Could not send report: SSL_connect returned=1 errno=0 state=error: certificate verify failed: [self signed certificate in certificate chain for /CN=Puppet Root CA: 3b5ff26c750018]
[ERROR], [2020-06-22 07:53:17], EXITING: [/opt/puppetlabs/puppet/bin/puppet agent --test --noop --tags no_such_tag --waitforcert 10] failed to execute properly.

Comment 4 Ewoud Kohl van Wijngaarden 2020-07-01 16:00:06 UTC
Yes, this certainly looks like that.

Can you confirm this is a Puppet 5 agent with a Puppet 6 server?

Comment 5 Lukáš Hellebrandt 2020-07-02 09:21:05 UTC
On server:
# rpm -qa | grep -e puppetserver -e puppet-agent
puppet-agent-oauth-0.5.1-3.el7sat.noarch
puppetserver-6.9.2-1.el7sat.noarch
puppet-agent-6.14.0-2.el7sat.x86_64

On client:
# rpm -qa | grep -e puppetserver -e puppet-agent
puppet-agent-5.5.17-1.el7sat.x86_64

Comment 7 Ewoud Kohl van Wijngaarden 2020-07-14 13:01:34 UTC
There are a few scenarios:

* Upgrading from Puppetserver 5 to Puppetserver 6
  * No intermediate CA is generated
  * Puppet 6 agents are no problem
  * Puppet 5 agents are no problem
* Installing Puppetserver 6
  * Intermediate CA is generated
  * Puppet 6 agents are no problem
  * Puppet 5 agents are a problem

The Intermediate CA is generated, following Puppet's best practices. If Puppet 5 agents are desired, there is a workaround. You can remove the SSL directory (rm -rf /etc/puppetlabs/puppet/ssl) and restart puppetserver (systemctl restart puppetserver). A new CA will generated without the intermediate CA. It should be obvious that everywhere where the previous CA was used now needs to be replaced. These are all the agents that were connected to the Puppetserver.

However, it is recommended to only use Puppet 6 agents on fresh installations. Puppet 5 upstream will go EOL in November this year, effectively 5 months from now.

Comment 8 Tomer Brisker 2020-08-04 06:42:19 UTC
Following comment #7, should this issue be converted to a documentation BZ or just a release note?

Comment 12 Mike McCune 2022-01-28 22:30:40 UTC
Upon review of our valid but aging backlog the Satellite Team has concluded that this Bugzilla does not meet the criteria for a resolution in the near term, and are planning to close in a month. This message may be a repeat of a previous update and the bug is again being considered to be closed. If you have any concerns about this, please contact your Red Hat Account team. Thank you.

Comment 13 Mike McCune 2022-03-18 19:44:22 UTC
Thank you for your interest in Red Hat Satellite. We have evaluated this request, and while we recognize that it is a valid request, we do not expect this to be implemented in the product in the foreseeable future. This is due to other priorities for the product, and not a reflection on the request itself. We are therefore closing this out as WONTFIX. If you have any concerns about this feel free to contact your Red Hat Account Team. Thank you.