Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1585754

Summary: Race condition when creating the candlepin keystore while running satellite-installer
Product: Red Hat Satellite Reporter: Barbora Vassova <bvassova>
Component: InstallationAssignee: satellite6-bugs <satellite6-bugs>
Status: CLOSED CURRENTRELEASE QA Contact: Katello QA List <katello-qa-list>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 6.2.14CC: brubisch, janarula, phess
Target Milestone: UnspecifiedKeywords: Patch
Target Release: Unused   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-06-14 17:01:54 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
patch file for Satellite 6.2.15's candlepin.pp none

Description Barbora Vassova 2018-06-04 15:48:46 UTC 
Description of problem:

While running satellite-installer it fails with following errors in the backtrace: 

At the begining, everything seems ok: 

[DEBUG 2018-06-01 16:26:29 main]  Exec[import client certificate into Candlepin keystore](provider=posix): Executing check 'keytool -list -keystore /etc/candlepin/certs/amqp/candlepin.jks -storepass <password> -alias katello-default-ca'
[DEBUG 2018-06-01 16:26:29 main]  Executing 'keytool -list -keystore /etc/candlepin/certs/amqp/candlepin.jks -storepass <password> -alias katello-default-ca'
[DEBUG 2018-06-01 16:26:29 main]  /Stage[main]/Certs::Candlepin/Exec[import client certificate into Candlepin keystore]/unless: keytool error: java.lang.Exception: Keystore file does not exist: /etc/candlepin/certs/amqp/candlepin.jks
[DEBUG 2018-06-01 16:26:29 main]  Exec[import client certificate into Candlepin keystore](provider=posix): Executing 'openssl pkcs12 -export -name amqp-client -in /etc/pki/katello/certs/java-client.crt -inkey /etc/pki/katello/private/java-client.key -out /tmp/keystore.p12 -passout file:/etc/pki/katello/keystore_password-file && keytool -importkeystore -destkeystore /etc/candlepin/certs/amqp/candlepin.jks -srckeystore /tmp/keystore.p12 -srcstoretype pkcs12 -alias amqp-client -storepass <password> -srcstorepass <password> -noprompt && rm /tmp/keystore.p12'
[DEBUG 2018-06-01 16:26:29 main]  Executing 'openssl pkcs12 -export -name amqp-client -in /etc/pki/katello/certs/java-client.crt -inkey /etc/pki/katello/private/java-client.key -out /tmp/keystore.p12 -passout file:/etc/pki/katello/keystore_password-file && keytool -importkeystore -destkeystore /etc/candlepin/certs/amqp/candlepin.jks -srckeystore /tmp/keystore.p12 -srcstoretype pkcs12 -alias amqp-client -storepass <password> -srcstorepass <password> -noprompt && rm /tmp/keystore.p12'

However keystore_password-file is suddently not being usable: 

[ WARN 2018-06-01 16:26:29 main]  /Stage[main]/Certs::Candlepin/Exec[import client certificate into Candlepin keystore]/returns: Can't open file /etc/pki/katello/keystore_password-file
[ WARN 2018-06-01 16:26:29 main]  /Stage[main]/Certs::Candlepin/Exec[import client certificate into Candlepin keystore]/returns: Error getting passwords
[ERROR 2018-06-01 16:26:29 main]  openssl pkcs12 -export -name amqp-client -in /etc/pki/katello/certs/java-client.crt -inkey /etc/pki/katello/private/java-client.key -out /tmp/keystore.p12 -passout file:/etc/pki/katello/keystore_password-file && keytool -importkeystore -destkeystore /etc/candlepin/certs/amqp/candlepin.jks -srckeystore /tmp/keystore.p12 -srcstoretype pkcs12 -alias amqp-client -storepass <password> -srcstorepass <password> -noprompt && rm /tmp/keystore.p12 returned 1 instead of one of [0]
[ INFO 2018-06-01 16:26:29 main] /usr/share/ruby/vendor_ruby/puppet/util/errors.rb:106:in `fail'
[ INFO 2018-06-01 16:26:29 main] /usr/share/ruby/vendor_ruby/puppet/type/exec.rb:160:in `sync'
[ INFO 2018-06-01 16:26:29 main] /usr/share/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:204:in `sync'
[ INFO 2018-06-01 16:26:29 main] /usr/share/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:128:in `sync_if_needed'
[ INFO 2018-06-01 16:26:29 main] /usr/share/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:89:in `block in perform_changes'
[ INFO 2018-06-01 16:26:29 main] /usr/share/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:88:in `each'
[ INFO 2018-06-01 16:26:29 main] /usr/share/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:88:in `perform_changes'
[ INFO 2018-06-01 16:26:29 main] /usr/share/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:20:in `evaluate'
[ INFO 2018-06-01 16:26:29 main] /usr/share/ruby/vendor_ruby/puppet/transaction.rb:204:in `apply'
[ INFO 2018-06-01 16:26:29 main] /usr/share/ruby/vendor_ruby/puppet/transaction.rb:217:in `eval_resource'
[ INFO 2018-06-01 16:26:29 main] /usr/share/ruby/vendor_ruby/puppet/transaction.rb:147:in `call'
[ INFO 2018-06-01 16:26:29 main] /usr/share/ruby/vendor_ruby/puppet/transaction.rb:147:in `block (2 levels) in evaluate'
[ INFO 2018-06-01 16:26:29 main] /usr/share/ruby/vendor_ruby/puppet/util.rb:335:in `block in thinmark'
[ INFO 2018-06-01 16:26:29 main] /usr/share/ruby/benchmark.rb:296:in `realtime'
[ INFO 2018-06-01 16:26:29 main] /usr/share/ruby/vendor_ruby/puppet/util.rb:334:in `thinmark'
[ INFO 2018-06-01 16:26:29 main] /usr/share/ruby/vendor_ruby/puppet/transaction.rb:147:in `block in evaluate'
[ INFO 2018-06-01 16:26:29 main] /usr/share/ruby/vendor_ruby/puppet/graph/relationship_graph.rb:118:in `traverse'
[ INFO 2018-06-01 16:26:29 main] /usr/share/ruby/vendor_ruby/puppet/transaction.rb:138:in `evaluate'
[ INFO 2018-06-01 16:26:29 main] /usr/share/gems/gems/kafo-0.7.6.1/modules/kafo_configure/lib/puppet/parser/functions/add_progress.rb:25:in `evaluate_with_trigger'
[ INFO 2018-06-01 16:26:29 main] /usr/share/ruby/vendor_ruby/puppet/resource/catalog.rb:169:in `block in apply'
[ INFO 2018-06-01 16:26:29 main] /usr/share/ruby/vendor_ruby/puppet/util/log.rb:149:in `with_destination'
[ INFO 2018-06-01 16:26:29 main] /usr/share/ruby/vendor_ruby/puppet/transaction/report.rb:112:in `as_logging_destination'
[ INFO 2018-06-01 16:26:29 main] /usr/share/gems/gems/kafo-0.7.6.1/modules/kafo_configure/lib/kafo/puppet/report_wrapper.rb:30:in `method_missing'
[ INFO 2018-06-01 16:26:29 main] /usr/share/ruby/vendor_ruby/puppet/resource/catalog.rb:168:in `apply'
[ INFO 2018-06-01 16:26:29 main] /usr/share/ruby/vendor_ruby/puppet/configurer.rb:120:in `block in apply_catalog'
[ INFO 2018-06-01 16:26:29 main] /usr/share/ruby/vendor_ruby/puppet/util.rb:161:in `block in benchmark'
[ INFO 2018-06-01 16:26:29 main] /usr/share/ruby/benchmark.rb:296:in `realtime'
[ INFO 2018-06-01 16:26:29 main] /usr/share/ruby/vendor_ruby/puppet/util.rb:160:in `benchmark'
[ INFO 2018-06-01 16:26:29 main] /usr/share/ruby/vendor_ruby/puppet/configurer.rb:119:in `apply_catalog'
[ INFO 2018-06-01 16:26:29 main] /usr/share/ruby/vendor_ruby/puppet/configurer.rb:227:in `run_internal'
[ INFO 2018-06-01 16:26:29 main] /usr/share/ruby/vendor_ruby/puppet/configurer.rb:134:in `block in run'
[ INFO 2018-06-01 16:26:29 main] /usr/share/ruby/vendor_ruby/puppet/context.rb:64:in `override'
[ INFO 2018-06-01 16:26:29 main] /usr/share/ruby/vendor_ruby/puppet.rb:246:in `override'
[ INFO 2018-06-01 16:26:29 main] /usr/share/ruby/vendor_ruby/puppet/configurer.rb:133:in `run'
[ INFO 2018-06-01 16:26:29 main] /usr/share/ruby/vendor_ruby/puppet/application/apply.rb:302:in `apply_catalog'
[ INFO 2018-06-01 16:26:29 main] /usr/share/ruby/vendor_ruby/puppet/application/apply.rb:236:in `block in main'
[ INFO 2018-06-01 16:26:29 main] /usr/share/ruby/vendor_ruby/puppet/context.rb:64:in `override'
[ INFO 2018-06-01 16:26:29 main] /usr/share/ruby/vendor_ruby/puppet.rb:246:in `override'
[ INFO 2018-06-01 16:26:29 main] /usr/share/ruby/vendor_ruby/puppet/application/apply.rb:198:in `main'
[ INFO 2018-06-01 16:26:29 main] /usr/share/ruby/vendor_ruby/puppet/application/apply.rb:159:in `run_command'
[ INFO 2018-06-01 16:26:29 main] /usr/share/ruby/vendor_ruby/puppet/application.rb:381:in `block (2 levels) in run'
[ INFO 2018-06-01 16:26:29 main] /usr/share/ruby/vendor_ruby/puppet/application.rb:507:in `plugin_hook'
[ INFO 2018-06-01 16:26:29 main] /usr/share/ruby/vendor_ruby/puppet/application.rb:381:in `block in run'
[ INFO 2018-06-01 16:26:29 main] /usr/share/ruby/vendor_ruby/puppet/util.rb:496:in `exit_on_fail'
[ INFO 2018-06-01 16:26:29 main] /usr/share/ruby/vendor_ruby/puppet/application.rb:381:in `run'
[ INFO 2018-06-01 16:26:29 main] /usr/share/ruby/vendor_ruby/puppet/util/command_line.rb:146:in `run'
[ INFO 2018-06-01 16:26:29 main] /usr/share/ruby/vendor_ruby/puppet/util/command_line.rb:92:in `execute'
[ INFO 2018-06-01 16:26:29 main] /usr/bin/puppet:8:in `<main>'
[ERROR 2018-06-01 16:26:29 main]  /Stage[main]/Certs::Candlepin/Exec[import client certificate into Candlepin keystore]/returns: change from notrun to 0 failed: openssl pkcs12 -export -name amqp-client -in /etc/pki/katello/certs/java-client.crt -inkey /etc/pki/katello/private/java-client.key -out /tmp/keystore.p12 -passout file:/etc/pki/katello/keystore_password-file && keytool -importkeystore -destkeystore /etc/candlepin/certs/amqp/candlepin.jks -srckeystore /tmp/keystore.p12 -srcstoretype pkcs12 -alias amqp-client -storepass <password> -srcstorepass <password> -noprompt && rm /tmp/keystore.p12 returned 1 instead of one of [0]
[ INFO 2018-06-01 16:26:29 main]  RESOURCE Exec[import client certificate into Candlepin keystore]
[DEBUG 2018-06-01 16:26:29 main]  Exec[import client certificate into Candlepin keystore](provider=posix): Executing check 'keytool -list -keystore /etc/candlepin/certs/amqp/candlepin.jks -storepass <password> -alias katello-default-ca'
[DEBUG 2018-06-01 16:26:29 main]  Executing 'keytool -list -keystore /etc/candlepin/certs/amqp/candlepin.jks -storepass <password> -alias katello-default-ca'
[DEBUG 2018-06-01 16:26:29 main]  /Stage[main]/Certs::Candlepin/Exec[import client certificate into Candlepin keystore]/unless: keytool error: java.lang.Exception: Keystore file does not exist: /etc/candlepin/certs/amqp/candlepin.jks
[DEBUG 2018-06-01 16:26:29 main]  Exec[import client certificate into Candlepin keystore](provider=posix): Executing 'openssl pkcs12 -export -name amqp-client -in /etc/pki/katello/certs/java-client.crt -inkey /etc/pki/katello/private/java-client.key -out /tmp/keystore.p12 -passout file:/etc/pki/katello/keystore_password-file && keytool -importkeystore -destkeystore /etc/candlepin/certs/amqp/candlepin.jks -srckeystore /tmp/keystore.p12 -srcstoretype pkcs12 -alias amqp-client -storepass <password> -srcstorepass <password> -noprompt && rm /tmp/keystore.p12'
[DEBUG 2018-06-01 16:26:29 main]  Executing 'openssl pkcs12 -export -name amqp-client -in /etc/pki/katello/certs/java-client.crt -inkey /etc/pki/katello/private/java-client.key -out /tmp/keystore.p12 -passout file:/etc/pki/katello/keystore_password-file && keytool -importkeystore -destkeystore /etc/candlepin/certs/amqp/candlepin.jks -srckeystore /tmp/keystore.p12 -srcstoretype pkcs12 -alias amqp-client -storepass <password> -srcstorepass <password> -noprompt && rm /tmp/keystore.p12'
[ WARN 2018-06-01 16:26:29 main]  /Stage[main]/Certs::Candlepin/Exec[import client certificate into Candlepin keystore]/returns: Can't open file /etc/pki/katello/keystore_password-file
[ WARN 2018-06-01 16:26:29 main]  /Stage[main]/Certs::Candlepin/Exec[import client certificate into Candlepin keystore]/returns: Error getting passwords
[ERROR 2018-06-01 16:26:29 main]  /Stage[main]/Certs::Candlepin/Exec[import client certificate into Candlepin keystore]: Failed to call refresh: openssl pkcs12 -export -name amqp-client -in /etc/pki/katello/certs/java-client.crt -inkey /etc/pki/katello/private/java-client.key -out /tmp/keystore.p12 -passout file:/etc/pki/katello/keystore_password-file && keytool -importkeystore -destkeystore /etc/candlepin/certs/amqp/candlepin.jks -srckeystore /tmp/keystore.p12 -srcstoretype pkcs12 -alias amqp-client -storepass <password> -srcstorepass <password> -noprompt && rm /tmp/keystore.p12 returned 1 instead of one of [0]
[ERROR 2018-06-01 16:26:29 main]  /Stage[main]/Certs::Candlepin/Exec[import client certificate into Candlepin keystore]: openssl pkcs12 -export -name amqp-client -in /etc/pki/katello/certs/java-client.crt -inkey /etc/pki/katello/private/java-client.key -out /tmp/keystore.p12 -passout file:/etc/pki/katello/keystore_password-file && keytool -importkeystore -destkeystore /etc/candlepin/certs/amqp/candlepin.jks -srckeystore /tmp/keystore.p12 -srcstoretype pkcs12 -alias amqp-client -storepass <password> -srcstorepass <password> -noprompt && rm /tmp/keystore.p12 returned 1 instead of one of [0]


But later is works:

[ WARN 2018-06-01 16:26:31 main]  /Stage[main]/Certs::Candlepin/File[/etc/pki/katello/keystore_password-file]/ensure: defined content as '{md5}<md>'
[ INFO 2018-06-01 16:26:31 main]  RESOURCE File[/etc/pki/katello/keystore_password-file]
[ INFO 2018-06-01 16:26:31 main]  /Stage[main]/Certs::Candlepin/File[/etc/pki/katello/keystore_password-file]: Scheduling refresh of Exec[candlepin-generate-ssl-keystore]

Installer finishes with:

[ERROR 2018-06-01 16:28:35 main] Errors encountered during run:
[ERROR 2018-06-01 16:28:35 main]  openssl pkcs12 -export -name amqp-client -in /etc/pki/katello/certs/java-client.crt -inkey /etc/pki/katello/private/java-client.key -out /tmp/keystore.p12 -passout file:/etc/pki/katello/keystore_password-file && keytool -importkeystore -destkeystore /etc/candlepin/certs/amqp/candlepin.jks -srckeystore /tmp/keystore.p12 -srcstoretype pkcs12 -alias amqp-client -storepass <password> -srcstorepass <password> -noprompt && rm /tmp/keystore.p12 returned 1 instead of one of [0]
[ERROR 2018-06-01 16:28:35 main]  /Stage[main]/Certs::Candlepin/Exec[import client certificate into Candlepin keystore]/returns: change from notrun to 0 failed: openssl pkcs12 -export -name amqp-client -in /etc/pki/katello/certs/java-client.crt -inkey /etc/pki/katello/private/java-client.key -out /tmp/keystore.p12 -passout file:/etc/pki/katello/keystore_password-file && keytool -importkeystore -destkeystore /etc/candlepin/certs/amqp/candlepin.jks -srckeystore /tmp/keystore.p12 -srcstoretype pkcs12 -alias amqp-client -storepass <password> -srcstorepass <password> -noprompt && rm /tmp/keystore.p12 returned 1 instead of one of [0]
[ERROR 2018-06-01 16:28:35 main]  /Stage[main]/Certs::Candlepin/Exec[import client certificate into Candlepin keystore]: Failed to call refresh: openssl pkcs12 -export -name amqp-client -in /etc/pki/katello/certs/java-client.crt -inkey /etc/pki/katello/private/java-client.key -out /tmp/keystore.p12 -passout file:/etc/pki/katello/keystore_password-file && keytool -importkeystore -destkeystore /etc/candlepin/certs/amqp/candlepin.jks -srckeystore /tmp/keystore.p12 -srcstoretype pkcs12 -alias amqp-client -storepass <password> -srcstorepass <password> -noprompt && rm /tmp/keystore.p12 returned 1 instead of one of [0]
[ERROR 2018-06-01 16:28:35 main]  /Stage[main]/Certs::Candlepin/Exec[import client certificate into Candlepin keystore]: openssl pkcs12 -export -name amqp-client -in /etc/pki/katello/certs/java-client.crt -inkey /etc/pki/katello/private/java-client.key -out /tmp/keystore.p12 -passout file:/etc/pki/katello/keystore_password-file && keytool -importkeystore -destkeystore /etc/candlepin/certs/amqp/candlepin.jks -srckeystore /tmp/keystore.p12 -srcstoretype pkcs12 -alias amqp-client -storepass <password> -srcstorepass <password> -noprompt && rm /tmp/keystore.p12 returned 1 instead of one of [0]
[ERROR 2018-06-01 16:28:35 main]  /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: Failed to call refresh: /usr/sbin/foreman-rake db:seed returned 1 instead of one of [0]
[ERROR 2018-06-01 16:28:35 main]  /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: /usr/sbin/foreman-rake db:seed returned 1 instead of one of [0]
[ERROR 2018-06-01 16:28:35 main]  /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[<hostname>]: Failed to call refresh: Proxy <hostname> cannot be registered (ApipieBindings::MissingArgumentsError: id): N/A
[ERROR 2018-06-01 16:28:35 main]  /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[<hostname>]: Proxy <hostname> cannot be registered (ApipieBindings::MissingArgumentsError: id): N/A
[DEBUG 2018-06-01 16:28:35 main] Cleaning /etc/foreman-installer/scenarios.d/d20180601-1633-1gl2mw6

Version-Release number of selected component (if applicable):
Satellite 6.2.15

Steps to Reproduce:
1. run the satellite-installer

Actual results:
Installer finishes with errors regarding keystore and missing keystore password file

Expected results:
Installer finishes without errors

Additional info:
This issue may be a clone of redmine bug #19734 https://projects.theforeman.org/issues/19734
Comment 1 Pablo Hess 2018-06-04 16:58:13 UTC 
I talked to engineering and confirmed Satellite 6.2.x does not have the patch applied but 6.3.x does.

Anyway, the upstream patch may be applied to Satellite 6.2.x code. The candlepin.pp manifest file is just slightly refactored so the change would be thus:

    6.2.15:
     
        } ~>
     
        file { "/usr/share/${candlepin::tomcat}/conf/keystore":
          ensure => link,
          target => $keystore,
          owner  => 'tomcat',
          group  => $::certs::group,
          notify => Service[$candlepin::tomcat],
        }
     
        Cert[$java_client_cert_name] ~>
        pubkey { $client_cert:
          key_pair => Cert[$java_client_cert_name],
        } ~>
        privkey { $client_key:
          key_pair => Cert[$java_client_cert_name],
        } ~>
     
     
    Update to:
     
        } ~>
        file { "/usr/share/${candlepin::tomcat}/conf/keystore":
          ensure => link,
          target => $keystore,
          owner  => 'tomcat',
          group  => $::certs::group,
          notify => Service[$candlepin::tomcat],
        } ->
        Cert[$java_client_cert_name] ~>
        pubkey { $client_cert:
          key_pair => Cert[$java_client_cert_name],
        } ~>
        privkey { $client_key:
          key_pair => Cert[$java_client_cert_name],
        } ~>
Comment 2 Pablo Hess 2018-06-04 20:53:20 UTC 
Created attachment 1447640 [details]
patch file for Satellite 6.2.15's candlepin.pp

Patch may be applied with:

# cd /usr/share/katello-installer-base/modules/certs/manifests
# patch -p1 < /path/to/candlepin-race-condition.patch

Then re-run satellite-installer.
Comment 5 Brad Buckingham 2018-06-14 17:01:54 UTC 
At this time, there are no plans to backport the fix to Satellite 6.2.z; therefore, closing this as 'current release'.