Bug 1399859

Summary: need a way to specify the ca_server when installing a puppet master capsule that is not a Puppet CA
Product: Red Hat Satellite Reporter: Ian Tewksbury <itewksbu>
Component: InstallationAssignee: satellite6-bugs <satellite6-bugs>
Status: CLOSED DUPLICATE QA Contact: Katello QA List <katello-qa-list>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 6.2.4CC: bbuckingham, bscalio, itewksbu, stbenjam
Target Milestone: Unspecified   
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-12-02 19:30:21 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Ian Tewksbury 2016-11-29 22:46:21 UTC
Description of problem:

When I attempt to install a capsule server with --foreman-proxy-puppetca set to false and --capsule-puppet set to true I get errors.

How reproducible:

Always.

Steps to Reproduce:

> satellite-installer --scenario capsule \
  --capsule-parent-fqdn                         "satellite6-master-0.rhc-lab.iad.redhat.com" \
  --foreman-proxy-register-in-foreman           "true" \
  --foreman-proxy-foreman-base-url              "https://satellite6-master-0.rhc-lab.iad.redhat.com" \
  --foreman-proxy-trusted-hosts                 "satellite6-master-0.rhc-lab.iad.redhat.com" \
  --foreman-proxy-trusted-hosts                 "sat-cap-content-1.rhc-lab.iad.redhat.com" \
  --foreman-proxy-oauth-consumer-key            "MKbWhrvnRgCX9TTMpCAZNPnALxBWMsYr" \
  --foreman-proxy-oauth-consumer-secret         "rooUFejJF2gHsF4KrsZpo9Gigjo74T9y" \
  --capsule-pulp-oauth-secret                   "n5NfA4AJF6cvgsMJEzgJGpk9QShSNFB3" \
  --capsule-certs-tar                           "~/sat-cap-content-1.rhc-lab.iad.redhat.com.tar" \
  --capsule-puppet-ca-proxy                     "https://satellite6-master-0.rhc-lab.iad.redhat.com:8140" \
  --foreman-proxy-puppetca                      "false"
  --capsule-puppet                              "true"
 Could not start Service[httpd]: Execution of '/usr/share/katello-installer-base/modules/service_wait/bin/service-wait start httpd' returned 1: Redirecting to /bin/systemctl start  httpd.service
 /Stage[main]/Apache::Service/Service[httpd]/ensure: change from stopped to running failed: Could not start Service[httpd]: Execution of '/usr/share/katello-installer-base/modules/service_wait/bin/service-wait start httpd' returned 1: Redirecting to /bin/systemctl start  httpd.service
Installing             Done                                               [100%] [.........................................................................................................................................................]
  Something went wrong! Check the log for ERROR-level output
  The full log is at /var/log/foreman-installer/capsule.log
[root@sat-cap-content-1 ~]# systemctl status httpd
● httpd.service - The Apache HTTP Server
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; disabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Tue 2016-11-29 17:40:19 EST; 23s ago
     Docs: man:httpd(8)
           man:apachectl(8)
  Process: 11762 ExecStop=/bin/kill -WINCH ${MAINPID} (code=exited, status=1/FAILURE)
  Process: 11760 ExecStart=/usr/sbin/httpd $OPTIONS -DFOREGROUND (code=exited, status=1/FAILURE)
 Main PID: 11760 (code=exited, status=1/FAILURE)

Nov 29 17:40:19 sat-cap-content-1.rhc-lab.iad.redhat.com systemd[1]: Starting The Apache HTTP Server...
Nov 29 17:40:19 sat-cap-content-1.rhc-lab.iad.redhat.com httpd[11760]: AH00526: Syntax error on line 34 of /etc/httpd/conf.d/25-puppet.conf:
Nov 29 17:40:19 sat-cap-content-1.rhc-lab.iad.redhat.com httpd[11760]: SSLCertificateFile: file '/var/lib/puppet/ssl/certs/sat-cap-content-1.rhc-lab.iad.redhat.com.pem' does not exist or is empty
Nov 29 17:40:19 sat-cap-content-1.rhc-lab.iad.redhat.com systemd[1]: httpd.service: main process exited, code=exited, status=1/FAILURE
Nov 29 17:40:19 sat-cap-content-1.rhc-lab.iad.redhat.com kill[11762]: kill: cannot find process ""
Nov 29 17:40:19 sat-cap-content-1.rhc-lab.iad.redhat.com systemd[1]: httpd.service: control process exited, code=exited status=1
Nov 29 17:40:19 sat-cap-content-1.rhc-lab.iad.redhat.com systemd[1]: Failed to start The Apache HTTP Server.
Nov 29 17:40:19 sat-cap-content-1.rhc-lab.iad.redhat.com systemd[1]: Unit httpd.service entered failed state.
Nov 29 17:40:19 sat-cap-content-1.rhc-lab.iad.redhat.com systemd[1]: httpd.service failed.
[root@sat-cap-content-1 ~]# ls /var/lib/puppet/ssl/certs
[root@sat-cap-content-1 ~]# 


Actual results:

Capsule attempts to use itself as the Puppet CA server even though it isn't one so the installer fails.

Expected results:

Need the installer to either:

a) use the provided --foreman-proxy-puppetca as the ca_server to request the puppet master certificate
b) provide another flag that can be used to specify the ca_server


Additional info:

This looks to be one of the errors described here, https://bugzilla.redhat.com/show_bug.cgi?id=1260973, but the issue does not seem to be resolved.

Work Arounds:

Option 1)

capsule# satellite-installer --scenario capsule \
  --capsule-parent-fqdn                         "satellite6-master-0.rhc-lab.iad.redhat.com" \
  --foreman-proxy-register-in-foreman           "true" \
  --foreman-proxy-foreman-base-url              "https://satellite6-master-0.rhc-lab.iad.redhat.com" \
  --foreman-proxy-trusted-hosts                 "satellite6-master-0.rhc-lab.iad.redhat.com" \
  --foreman-proxy-trusted-hosts                 "sat-cap-content-1.rhc-lab.iad.redhat.com" \
  --foreman-proxy-oauth-consumer-key            "MKbWhrvnRgCX9TTMpCAZNPnALxBWMsYr" \
  --foreman-proxy-oauth-consumer-secret         "rooUFejJF2gHsF4KrsZpo9Gigjo74T9y" \
  --capsule-pulp-oauth-secret                   "n5NfA4AJF6cvgsMJEzgJGpk9QShSNFB3" \
  --capsule-certs-tar                           "~/sat-cap-content-1.rhc-lab.iad.redhat.com.tar" \
  --capsule-puppet-ca-proxy                     "https://satellite6-master-0.rhc-lab.iad.redhat.com:8140" \
  --foreman-proxy-puppetca                      "false"
  --capsule-puppet                              "true"
 Could not start Service[httpd]: Execution of '/usr/share/katello-installer-base/modules/service_wait/bin/service-wait start httpd' returned 1: Redirecting to /bin/systemctl start  httpd.service
 /Stage[main]/Apache::Service/Service[httpd]/ensure: change from stopped to running failed: Could not start Service[httpd]: Execution of '/usr/share/katello-installer-base/modules/service_wait/bin/service-wait start httpd' returned 1: Redirecting to /bin/systemctl start  httpd.service
Installing             Done                                               [100%] [.........................................................................................................................................................]
  Something went wrong! Check the log for ERROR-level output
  The full log is at /var/log/foreman-installer/capsule.log

capsule# vi /etc/puppet/puppet.conf
ca_server = satellite6-master-0.rhc-lab.iad.redhat.com

capsule# puppet agent -t
Info: Caching certificate for ca
Info: csr_attributes file loading from /etc/puppet/csr_attributes.yaml
Info: Creating a new SSL certificate request for sat-cap-content-1.rhc-lab.iad.redhat.com
Info: Certificate Request fingerprint (SHA256): D1:74:59:6C:B4:4F:6D:3E:95:10:27:F2:9C:37:F5:A5:11:84:89:06:88:9E:C9:9A:DE:E5:B2:CE:22:3A:0B:B1
Info: Caching certificate for sat-cap-content-1.rhc-lab.iad.redhat.com
Info: Caching certificate_revocation_list for ca
Info: Caching certificate for ca
Warning: Unable to fetch my node definition, but the agent run will continue:
Warning: Connection refused - connect(2)
Info: Retrieving pluginfacts
Error: /File[/var/lib/puppet/facts.d]: Failed to generate additional resources using 'eval_generate': Connection refused - connect(2)
Error: /File[/var/lib/puppet/facts.d]: Could not evaluate: Could not retrieve file metadata for puppet://sat-cap-content-1.rhc-lab.iad.redhat.com/pluginfacts: Connection refused - connect(2)
Info: Retrieving plugin
Error: /File[/var/lib/puppet/lib]: Failed to generate additional resources using 'eval_generate': Connection refused - connect(2)
Error: /File[/var/lib/puppet/lib]: Could not evaluate: Could not retrieve file metadata for puppet://sat-cap-content-1.rhc-lab.iad.redhat.com/plugins: Connection refused - connect(2)
Error: Could not retrieve catalog from remote server: Connection refused - connect(2)
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run
Error: Could not send report: Connection refused - connect(2)


capsule# satellite-installer --scenario capsule \
  --capsule-parent-fqdn                         "satellite6-master-0.rhc-lab.iad.redhat.com" \
  --foreman-proxy-register-in-foreman           "true" \
  --foreman-proxy-foreman-base-url              "https://satellite6-master-0.rhc-lab.iad.redhat.com" \
  --foreman-proxy-trusted-hosts                 "satellite6-master-0.rhc-lab.iad.redhat.com" \
  --foreman-proxy-trusted-hosts                 "sat-cap-content-1.rhc-lab.iad.redhat.com" \
  --foreman-proxy-oauth-consumer-key            "MKbWhrvnRgCX9TTMpCAZNPnALxBWMsYr" \
  --foreman-proxy-oauth-consumer-secret         "rooUFejJF2gHsF4KrsZpo9Gigjo74T9y" \
  --capsule-pulp-oauth-secret                   "n5NfA4AJF6cvgsMJEzgJGpk9QShSNFB3" \
  --capsule-certs-tar                           "~/sat-cap-content-1.rhc-lab.iad.redhat.com.tar" \
  --capsule-puppet-ca-proxy                     "https://satellite6-master-0.rhc-lab.iad.redhat.com:8140" \
  --foreman-proxy-puppetca                      "false"
  --capsule-puppet                              "true"
Installing             Done                                               [100%] [.........................................................................................................................................................]
  Success!
  The full log is at /var/log/foreman-installer/capsule.log

Comment 2 Stephen Benjamin 2016-12-02 19:30:21 UTC
Hi,

The external puppet ca proxying doesn't work. That's covered by BZ1233302, so marking this a dupe.  We only have partial support for it and looks like it doesn't work with the capsule installer.  

Even if you get the capsule to install correctly with how you're trying to do things, you'll still need to configure provisioned hosts to hit the capsule for puppet but the satellite for the CA, which might not be possible if you have segregated networks.  The reason is, Satellite needs some awareness of the concept so it can correctly add the autosign.conf entries to the right place (covered by that BZ I mentioned).

The Puppet CA's are a bit of a problem child in Satellite, the best option is just leave it alone to how we have it and use them as independent CA's.  I realize this links a client with only a single capsule.  There is a ref arch for HA if you need that, although it relies on cluster suite.

Just a little more info in case you're interested, there's a bunch of history here we need to correct :-\.  Since the beginning the Puppet CA was left alone and outside of Satellite PKI. It's a pain, and something we're looking to fix  -- the CA proxy is one way, but it's likely not the one we'll recommend anyway.

It'll get fixed at some point in case you want to use it, but I'm thinking the default approach will be to turn off Puppet CA everywhere entirely and use the consumer certificates from subscription-manager, that's covered by https://bugzilla.redhat.com/show_bug.cgi?id=1348660.

*** This bug has been marked as a duplicate of bug 1233302 ***

Comment 3 Ian Tewksbury 2016-12-02 19:49:13 UTC
@stephen,

Thanks for the info.

As I said in my original report I was able to get it working using the work around I posted. If you were interested in seeing that setup at all let me know.

Comment 4 Stephen Benjamin 2016-12-02 20:07:12 UTC
Ah I see now, we just need to make sure we generate the puppet certificate before running the installer, I added some notes to the BZ1233302 based on your findings, thanks!