Bug 1195463

Summary: getting "Could not evaluate: Could not load data from https://<fqdn>" during `katello-installer --upgrade`
Product: Red Hat Satellite Reporter: Jan Hutař <jhutar>
Component: InstallationAssignee: Stephen Benjamin <stbenjam>
Status: CLOSED DUPLICATE QA Contact: Katello QA List <katello-qa-list>
Severity: high Docs Contact:
Priority: unspecified    
Version: 6.1.0CC: bbuckingham, cwelton
Target Milestone: UnspecifiedKeywords: Triaged
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-05-11 20:35:21 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1195450    

Description Jan Hutař 2015-02-23 21:37:13 UTC
Description of problem:
I'm getting "Could not evaluate: Could not load data from https://<fqdn>" during `katello-installer --upgrade`.


Version-Release number of selected component (if applicable):
Upgrading 6.0.4 to Satellite-6.1.0-RHEL-6-20150220.0-Satellite-x86_64
+ upgraded to packages to workaround bug 1194997 (candlepin-0.9.43-1.el6 + candlepin-selinux-0.9.43-1.el6 + candlepin-tomcat6-0.9.43-1.el6)


How reproducible:
1 of 1 system


Steps to Reproduce:
1. Install 6.0.4
2. Upgrade packages to 6.1
3. Upgrade candlepin packages to workaroud bug 1194997
4. Workaround bug 1195262
5. # katello-installer --upgrade


Actual results:
# katello-installer --upgrade
Upgrading...
Upgrade Step: migrate_pulp...
Upgrade Step: migrate_candlepin...
Upgrade Step: migrate_foreman...
Upgrade Step: Running installer...
 Could not start Service[foreman-tasks]: Execution of '/sbin/service foreman-tasks start' returned 10: Starting foreman-tasks: Some processes of foreman-tasks is already running. Some not. Please stop foreman-tasks first.[FAILED]
 /Stage[main]/Foreman::Plugin::Tasks/Service[foreman-tasks]/ensure: change from stopped to running failed: Could not start Service[foreman-tasks]: Execution of '/sbin/service foreman-tasks start' returned 10: Starting foreman-tasks: Some processes of foreman-tasks is already running. Some not. Please stop foreman-tasks first.[FAILED]
 /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[<fqdn>]: Could not evaluate: Could not load data from https://<fqdn>
Installing             Done                                               [100%] [.........................................................................................................................................................]
Upgrade Step: db:seed...
Upgrade Step: Running errata import task (this may take a while)...
  Something went wrong! Check the log for ERROR-level output
  The full log is at /var/log/katello-installer/katello-installer.log


Expected results:
Should pass.


Additional info:
Services seems to be OK (except for foreman-tasks) before step 5 (this is Xth attempt).
Relevant part of the log is:
# grep -B 3 -A 3 '\[ERROR' /var/log/katello-installer/katello-installer.log
[DEBUG 2015-02-23 16:25:41 main]  Executing '/sbin/service foreman-tasks status'
[DEBUG 2015-02-23 16:25:41 main]  Executing '/sbin/chkconfig foreman-tasks'
[DEBUG 2015-02-23 16:25:41 main]  Executing '/sbin/service foreman-tasks start'
[ERROR 2015-02-23 16:25:41 main]  Could not start Service[foreman-tasks]: Execution of '/sbin/service foreman-tasks start' returned 10: Starting foreman-tasks: Some processes of foreman-tasks is already running. Some not. Please stop foreman-tasks first.[FAILED]
[ INFO 2015-02-23 16:25:41 main] /usr/lib/ruby/site_ruby/1.8/puppet/util/execution.rb:197:in `execute'
[ INFO 2015-02-23 16:25:41 main] /usr/lib/ruby/site_ruby/1.8/puppet/provider.rb:115:in `execute'
[ INFO 2015-02-23 16:25:41 main] /usr/lib/ruby/site_ruby/1.8/puppet/provider/service/service.rb:25:in `texecute'
--
[ INFO 2015-02-23 16:25:41 main] /usr/lib/ruby/site_ruby/1.8/puppet/util/command_line.rb:137:in `run'
[ INFO 2015-02-23 16:25:41 main] /usr/lib/ruby/site_ruby/1.8/puppet/util/command_line.rb:91:in `execute'
[ INFO 2015-02-23 16:25:41 main] /usr/bin/puppet:8
[ERROR 2015-02-23 16:25:41 main]  /Stage[main]/Foreman::Plugin::Tasks/Service[foreman-tasks]/ensure: change from stopped to running failed: Could not start Service[foreman-tasks]: Execution of '/sbin/service foreman-tasks start' returned 10: Starting foreman-tasks: Some processes of foreman-tasks is already running. Some not. Please stop foreman-tasks first.[FAILED]
[ INFO 2015-02-23 16:25:41 main]  RESOURCE Service[foreman-tasks]
[ INFO 2015-02-23 16:25:41 main]  RESOURCE Package[ruby193-rubygem-foreman_bootdisk]
[ INFO 2015-02-23 16:25:41 main]  RESOURCE Foreman::Plugin[bootdisk]
--
[DEBUG 2015-02-23 16:26:07 main]  /Stage[main]/Certs::Foreman/Exec[foreman_certs_config]/unless: /etc/foreman/client_cert.pem
[DEBUG 2015-02-23 16:26:07 main]  /Stage[main]/Certs::Foreman/Exec[foreman_certs_config]/unless: /etc/foreman/client_key.pem
[ INFO 2015-02-23 16:26:07 main]  RESOURCE Exec[foreman_certs_config]
[ERROR 2015-02-23 16:26:36 main]  /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[<fqdn>]: Could not evaluate: Could not load data from https://<fqdn>
[ INFO 2015-02-23 16:26:36 main]  - is your server down?
[ INFO 2015-02-23 16:26:36 main]  - was rake apipie:cache run when using apipie cache? (typical production settings)
[ INFO 2015-02-23 16:26:36 main] /usr/lib/ruby/gems/1.8/gems/apipie-bindings-0.0.11/lib/apipie_bindings/api.rb:264:in `retrieve_apidoc'
--
[DEBUG 2015-02-23 16:27:33 main] Hook /usr/share/katello-installer/hooks/post/10-post_install.rb returned nil
[ INFO 2015-02-23 16:27:33 main] All hooks in group post finished
[DEBUG 2015-02-23 16:27:33 main] Exit with status code: 6 (signal was 6)
[ERROR 2015-02-23 16:27:33 main] Repeating errors encountered during run:
[ERROR 2015-02-23 16:27:33 main]  Could not start Service[foreman-tasks]: Execution of '/sbin/service foreman-tasks start' returned 10: Starting foreman-tasks: Some processes of foreman-tasks is already running. Some not. Please stop foreman-tasks first.[FAILED]
[ERROR 2015-02-23 16:27:33 main]  /Stage[main]/Foreman::Plugin::Tasks/Service[foreman-tasks]/ensure: change from stopped to running failed: Could not start Service[foreman-tasks]: Execution of '/sbin/service foreman-tasks start' returned 10: Starting foreman-tasks: Some processes of foreman-tasks is already running. Some not. Please stop foreman-tasks first.[FAILED]
[ERROR 2015-02-23 16:27:33 main]  /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[<fqdn>]: Could not evaluate: Could not load data from https://<fqdn>
[DEBUG 2015-02-23 16:27:33 main] Cleaning /etc/katello-installer/d20150223-20522-1ag9r7d
[DEBUG 2015-02-23 16:27:33 main] Cleaning /tmp/default_values.yaml

Comment 1 Jan Hutař 2015-02-23 22:10:47 UTC
Then I have fixed problem with foreman-tasks, this is the output:

# katello-installer --upgrade
Upgrading...
Upgrade Step: migrate_pulp...
Upgrade Step: migrate_candlepin...
Upgrade Step: migrate_foreman...
Upgrade Step: Running installer...
 /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[<fqdn>]: Could not evaluate: Could not load data from https://<fqdn>
 /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[<fqdn>]: Failed to call refresh: Could not load data from https://<fqdn>
 /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[<fqdn>]: Could not load data from https://<fqdn>
Installing             Done                                               [100%] [.........................................................................................................................................................]
Upgrade Step: db:seed...
Upgrade Step: Running errata import task (this may take a while)...
  Something went wrong! Check the log for ERROR-level output
  The full log is at /var/log/katello-installer/katello-installer.log
# grep -B 3 -A 3 '\[ERROR' /var/log/katello-installer/katello-installer.log | sed "s/$(hostname)/<fqdn>/g"
[DEBUG 2015-02-23 17:00:29 main]  /Stage[main]/Certs::Foreman/Exec[foreman_certs_config]/unless: /etc/foreman/client_cert.pem
[DEBUG 2015-02-23 17:00:29 main]  /Stage[main]/Certs::Foreman/Exec[foreman_certs_config]/unless: /etc/foreman/client_key.pem
[ INFO 2015-02-23 17:00:29 main]  RESOURCE Exec[foreman_certs_config]
[ERROR 2015-02-23 17:00:58 main]  /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[<fqdn>]: Could not evaluate: Could not load data from https://<fqdn>
[ INFO 2015-02-23 17:00:58 main]  - is your server down?
[ INFO 2015-02-23 17:00:58 main]  - was rake apipie:cache run when using apipie cache? (typical production settings)
[ INFO 2015-02-23 17:00:58 main] /usr/lib/ruby/gems/1.8/gems/apipie-bindings-0.0.11/lib/apipie_bindings/api.rb:264:in `retrieve_apidoc'
--
[ INFO 2015-02-23 17:00:58 main] /usr/lib/ruby/site_ruby/1.8/puppet/util/command_line.rb:91:in `execute'
[ INFO 2015-02-23 17:00:58 main] /usr/bin/puppet:8
[ INFO 2015-02-23 17:00:58 main]  RESOURCE Foreman_smartproxy[<fqdn>]
[ERROR 2015-02-23 17:01:26 main]  /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[<fqdn>]: Failed to call refresh: Could not load data from https://<fqdn>
[ INFO 2015-02-23 17:01:26 main]  - is your server down?
[ INFO 2015-02-23 17:01:26 main]  - was rake apipie:cache run when using apipie cache? (typical production settings)
[ERROR 2015-02-23 17:01:26 main]  /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[<fqdn>]: Could not load data from https://<fqdn>
[ INFO 2015-02-23 17:01:26 main]  - is your server down?
[ INFO 2015-02-23 17:01:26 main]  - was rake apipie:cache run when using apipie cache? (typical production settings)
[ INFO 2015-02-23 17:01:26 main] /usr/lib/ruby/gems/1.8/gems/apipie-bindings-0.0.11/lib/apipie_bindings/api.rb:264:in `retrieve_apidoc'
--
[DEBUG 2015-02-23 17:02:24 main] Hook /usr/share/katello-installer/hooks/post/10-post_install.rb returned nil
[ INFO 2015-02-23 17:02:24 main] All hooks in group post finished
[DEBUG 2015-02-23 17:02:24 main] Exit with status code: 6 (signal was 6)
[ERROR 2015-02-23 17:02:24 main] Repeating errors encountered during run:
[ERROR 2015-02-23 17:02:24 main]  /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[<fqdn>]: Could not evaluate: Could not load data from https://<fqdn>
[ERROR 2015-02-23 17:02:24 main]  /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[<fqdn>]: Failed to call refresh: Could not load data from https://<fqdn>
[ERROR 2015-02-23 17:02:24 main]  /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[<fqdn>]: Could not load data from https://<fqdn>
[DEBUG 2015-02-23 17:02:24 main] Cleaning /etc/katello-installer/d20150223-25461-qg7g08
[DEBUG 2015-02-23 17:02:24 main] Cleaning /tmp/default_values.yaml

Comment 2 RHEL Program Management 2015-02-24 08:15:57 UTC
Since this issue was entered in Red Hat Bugzilla, the release flag has been
set to ? to ensure that it is properly evaluated for this release.

Comment 5 Stephen Benjamin 2015-05-11 20:35:21 UTC
I'm going to mark this as a duplicate of BZ1205737.  A number of improvements were made to how we handle services during the upgrade, so I think this will resolve your problem.

At the very least, the preflight check of the upgrade will now detect the bad state of foreman-tasks and bail out with a more helpful message, giving the user the opportunity to fix it before continuing.

*** This bug has been marked as a duplicate of bug 1205737 ***