Bug 1501980

Summary: Hammer errors out with certificate errors when using custom certs
Product: Red Hat Satellite Reporter: Johan Swensson <jswensso>
Component: InstallationAssignee: Stephen Benjamin <stbenjam>
Status: CLOSED ERRATA QA Contact: Nikhil Kathole <nkathole>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 6.3.0CC: akarsale, bbuckingham, chrobert, cmarinea, dhlavacd, ehelms, kgaikwad, lzap, mhulan, mmccune, nkathole, rabajaj, sabnave, stbenjam, sthirugn, tstrachota
Target Milestone: UnspecifiedKeywords: Triaged
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: katello-installer-base-3.4.5.20-1,foreman-installer-1.15.6.7-1 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-02-21 16:54:17 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1122832, 1533259    

Description Johan Swensson 2017-10-13 15:56:58 UTC 
Description of problem:
When deploying Satellite 6.3 snap 19 with a custom certificate hammer errors out with every command until :ssl_ca_file: is configured to point to the ca bundle.

Version-Release number of selected component (if applicable):
6.3 snap19

How reproducible:
Easy

Steps to Reproduce:
1. Install Satellite with custom certificates
2. run hammer with out arguments
3.

Actual results:
# hammer
Could not load the API description from the server: SSL certificate verification failed
Make sure you configured the correct URL and have the server's CA certificate installed on your system.

The following configuration option were used for the SSL connection:
  ssl_ca_file = /etc/pki/katello/certs/katello-default-ca.crt

Make sure the location contains an unexpired and valid CA certificate for https://sat63-snap19.example.com

Warning: An error occured while loading module hammer_cli_csv
Could not load the API description from the server: SSL certificate verification failed
Make sure you configured the correct URL and have the server's CA certificate installed on your system.

The following configuration option were used for the SSL connection:
  ssl_ca_file = /etc/pki/katello/certs/katello-default-ca.crt

Make sure the location contains an unexpired and valid CA certificate for https://sat63-snap19.example.com

Warning: An error occured while loading module hammer_cli_foreman
Could not load the API description from the server: SSL certificate verification failed
Make sure you configured the correct URL and have the server's CA certificate installed on your system.

The following configuration option were used for the SSL connection:
  ssl_ca_file = /etc/pki/katello/certs/katello-default-ca.crt

Make sure the location contains an unexpired and valid CA certificate for https://sat63-snap19.example.com

Warning: An error occured while loading module hammer_cli_foreman_bootdisk
Could not load the API description from the server: SSL certificate verification failed
Make sure you configured the correct URL and have the server's CA certificate installed on your system.

The following configuration option were used for the SSL connection:
  ssl_ca_file = /etc/pki/katello/certs/katello-default-ca.crt

Make sure the location contains an unexpired and valid CA certificate for https://sat63-snap19.example.com

Warning: An error occured while loading module hammer_cli_foreman_docker
Could not load the API description from the server: SSL certificate verification failed
Make sure you configured the correct URL and have the server's CA certificate installed on your system.

The following configuration option were used for the SSL connection:
  ssl_ca_file = /etc/pki/katello/certs/katello-default-ca.crt

Make sure the location contains an unexpired and valid CA certificate for https://sat63-snap19.example.com

Warning: An error occured while loading module hammer_cli_foreman_openscap
Could not load the API description from the server: SSL certificate verification failed
Make sure you configured the correct URL and have the server's CA certificate installed on your system.

The following configuration option were used for the SSL connection:
  ssl_ca_file = /etc/pki/katello/certs/katello-default-ca.crt

Make sure the location contains an unexpired and valid CA certificate for https://sat63-snap19.example.com

Warning: An error occured while loading module hammer_cli_foreman_remote_execution
Warning: An error occured while loading module hammer_cli_foreman_tasks
Could not load the API description from the server: SSL certificate verification failed
Make sure you configured the correct URL and have the server's CA certificate installed on your system.

The following configuration option were used for the SSL connection:
  ssl_ca_file = /etc/pki/katello/certs/katello-default-ca.crt

Make sure the location contains an unexpired and valid CA certificate for https://sat63-snap19.example.com

Warning: An error occured while loading module hammer_cli_foreman_virt_who_configure
Could not load the API description from the server: SSL certificate verification failed
Make sure you configured the correct URL and have the server's CA certificate installed on your system.

The following configuration option were used for the SSL connection:
  ssl_ca_file = /etc/pki/katello/certs/katello-default-ca.crt

Make sure the location contains an unexpired and valid CA certificate for https://sat63-snap19.example.com

Warning: An error occured while loading module hammer_cli_katello


Expected results:
Hammer should work out of the box even when using custom certificates.

Additional info:
This was not needed in 6.2, so I'm not sure if this is something the installer does differently or what's going on but when creating a ~/.hammer/cli_config.yml with the following content it works:

:ssl:
  :ssl_ca_file: '/root/ca-chain.pem'


Satellite was installed with the following:  satellite-installer --scenario satellite --certs-server-cert "/root/sat63-snap19.example.com.crt"                      --certs-server-cert-req "/root/fake.csr" --certs-server-key "/root/sat63-snap19.example.com.key" --certs-server-ca-cert "/root/ca-chain.pem" --foreman-admin-password redhat123 --foreman-initial-organization "testday" --foreman-proxy-tftp true

And output of katello-certs-check:
Checking expiration of certificate: [OK]
Checking expiration of CA bundle: [OK]
Validating the certificate subject= /C=SE/ST=Stockholm/O=opuk lab/OU=opuk lab intermediate/CN=sat63-snap19.example.com/emailAddress=root
Checking to see if the private key matches the certificate: [OK]
Checking ca bundle against the cert file: [OK]
Checking for non ascii characters[OK]
Comment 2 Brad Buckingham 2017-11-08 15:07:29 UTC 
*** Bug 1508621 has been marked as a duplicate of this bug. ***
Comment 3 Brad Buckingham 2017-11-08 15:08:20 UTC 
Moving to Installer based upon https://bugzilla.redhat.com/show_bug.cgi?id=1508621#c0
Comment 4 Mike McCune 2017-11-10 14:28:44 UTC 
*** WORKAROUND ***


This breaks due to /etc/hammer/cli.modules.d/foreman.yml being configured to have ssl_ca_file pointed at /etc/pki/katello/certs/katello-default-ca.crt. This certificate is not the CA being used by the Foreman webserver when custom certificates are being used. Rather, this should be configured to point at:

/etc/pki/katello/certs/katello-server-ca.crt

edit /etc/hammer/cli.modules.d/foreman.yml and set the ca to the above file
Comment 6 Lukas Zapletal 2017-12-11 12:54:50 UTC 
*** Bug 1520476 has been marked as a duplicate of this bug. ***
Comment 8 Stephen Benjamin 2018-01-09 14:33:49 UTC 
Created redmine issue http://projects.theforeman.org/issues/22196 from this bug
Comment 9 Satellite Program 2018-01-09 15:11:28 UTC 
Upstream bug assigned to stbenjam
Comment 10 Satellite Program 2018-01-09 15:11:34 UTC 
Upstream bug assigned to stbenjam
Comment 11 Satellite Program 2018-01-10 11:11:44 UTC 
Moving this bug to POST for triage into Satellite 6 since the upstream issue http://projects.theforeman.org/issues/22196 has been resolved.
Comment 12 Satellite Program 2018-01-15 11:11:38 UTC 
Moving this bug to POST for triage into Satellite 6 since the upstream issue http://projects.theforeman.org/issues/22196 has been resolved.
Comment 15 Nikhil Kathole 2018-02-06 16:09:31 UTC 
VERIFIED

Version tested:
Satellite 6.3 snap 35

Hammer works correctly with satellite using custom certs.
Comment 17 Satellite Program 2018-02-21 16:54:17 UTC 
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA.
> 
> For information on the advisory, and where to find the updated files, follow the link below.
> 
> If the solution does not work for you, open a new bug report.
> 
> https://access.redhat.com/errata/RHSA-2018:0336