Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1245998 - After upgrade[sat6.0.8 -> sat6.1.1 ], capsule/katello-installer.log throws error on checking "httpd status", though upgrade is completed successfully
Summary: After upgrade[sat6.0.8 -> sat6.1.1 ], capsule/katello-installer.log throws er...
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Upgrades
Version: 6.1.0
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: Unspecified
Assignee: satellite6-bugs
QA Contact: Sachin Ghai
URL:
Whiteboard:
Depends On:
Blocks: 1195450
TreeView+ depends on / blocked
 
Reported: 2015-07-23 09:30 UTC by Sachin Ghai
Modified: 2017-01-11 19:44 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-01-11 19:44:04 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
katello-service status shows all services are 'active' without any error. (12.21 KB, text/plain)
2015-07-23 09:31 UTC, Sachin Ghai 		no flags Details
foreman-debug from capsule (274.15 KB, application/x-xz)
2015-07-23 09:34 UTC, Sachin Ghai 		no flags Details
foreman-debug logs from sat server (731.99 KB, application/x-xz)
2015-07-23 09:38 UTC, Sachin Ghai 		no flags Details
View All

Description Sachin Ghai 2015-07-23 09:30:32 UTC 
Description of problem:
I was trying capsule-upgrade from sat6.0.8 to sat6.1.1 GA snap14 on rhel71. Upgrade completed successfully. No error on stdout. However when I checked the capsule-installer.log, it ends with below error:

-----------
[ INFO 2015-07-23 05:13:28 main] Upgrade Step: restart_services...
[DEBUG 2015-07-23 05:13:41 main] Redirecting to /bin/systemctl restart  qpidd.service

Redirecting to /bin/systemctl restart  qdrouterd.service

Redirecting to /bin/systemctl restart  pulp_workers.service

Redirecting to /bin/systemctl restart  foreman-proxy.service

Redirecting to /bin/systemctl restart  pulp_resource_manager.service

Redirecting to /bin/systemctl restart  mongod.service

Redirecting to /bin/systemctl restart  pulp_celerybeat.service

Redirecting to /bin/systemctl restart  httpd.service

Success!

[ INFO 2015-07-23 05:13:41 main] Katello upgrade completed!
[DEBUG 2015-07-23 05:13:41 main] Hook /usr/share/katello-installer/hooks/post/30-upgrade.rb returned [<Logging::Logger:0x1ab5dc8 name="main">, <Logging::Logger:0x1aba0e4 name="fatal">]
[ INFO 2015-07-23 05:13:41 main] All hooks in group post finished
[DEBUG 2015-07-23 05:13:41 main] Exit with status code: 2 (signal was 2)
[ERROR 2015-07-23 05:13:41 main] Repeating errors encountered during run:
[ERROR 2015-07-23 05:13:41 main] Redirecting to /bin/systemctl status  httpd.service
httpd.service - The Apache HTTP Server
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled)
   Active: inactive (dead) since Thu 2015-07-23 04:54:45 EDT; 17min ago
 Main PID: 2086 (code=exited, status=0/SUCCESS)
   Status: "Total requests: 0; Current requests/sec: 0; Current traffic:   0 B/sec"
   CGroup: /system.slice/httpd.service

Jul 23 04:54:40 cloud-qe-20.idmqe.lab.eng.bos.redhat.com pulp[2125]: qpid.messaging:WARNING: recoverable error[attempt 3]: [Errno 111] Connection refused
Jul 23 04:54:40 cloud-qe-20.idmqe.lab.eng.bos.redhat.com pulp[2125]: qpid.messaging:WARNING: sleeping 4 seconds
Jul 23 04:54:40 cloud-qe-20.idmqe.lab.eng.bos.redhat.com pulp[2125]: qpid.messaging:WARNING: trying: cloud-qe-20.idmqe.lab.eng.bos.redhat.com:5671
Jul 23 04:54:40 cloud-qe-20.idmqe.lab.eng.bos.redhat.com pulp[2125]: qpid.messaging:WARNING: recoverable error[attempt 3]: [Errno 111] Connection refused
Jul 23 04:54:40 cloud-qe-20.idmqe.lab.eng.bos.redhat.com pulp[2125]: qpid.messaging:WARNING: sleeping 4 seconds
Jul 23 04:54:41 cloud-qe-20.idmqe.lab.eng.bos.redhat.com systemd[1]: Stopping The Apache HTTP Server...
Jul 23 04:54:45 cloud-qe-20.idmqe.lab.eng.bos.redhat.com pulp[2125]: qpid.messaging:WARNING: trying: cloud-qe-20.idmqe.lab.eng.bos.redhat.com:5671
Jul 23 04:54:45 cloud-qe-20.idmqe.lab.eng.bos.redhat.com pulp[2125]: qpid.messaging:WARNING: recoverable error[attempt 4]: [Errno 111] Connection refused
Jul 23 04:54:45 cloud-qe-20.idmqe.lab.eng.bos.redhat.com pulp[2125]: qpid.messaging:WARNING: sleeping 8 seconds
Jul 23 04:54:45 cloud-qe-20.idmqe.lab.eng.bos.redhat.com systemd[1]: Stopped The Apache HTTP Server.
-----------------


On manually, however httpd status is 'ok' when checking the httpd status after upgrade.


Version-Release number of selected component (if applicable):
sat6.1.1 GA snap14

How reproducible:


Steps to Reproduce:
1. capsule-installer --upgrade --certs-tar <tarball>
2.
3.

Actual results:
[ INFO 2015-07-23 05:13:41 main] Katello upgrade completed!
[DEBUG 2015-07-23 05:13:41 main] Hook /usr/share/katello-installer/hooks/post/30-upgrade.rb returned [<Logging::Logger:0x1ab5dc8 name="main">, <Logging::Logger:0x1aba0e4 name="fatal">]
[ INFO 2015-07-23 05:13:41 main] All hooks in group post finished
[DEBUG 2015-07-23 05:13:41 main] Exit with status code: 2 (signal was 2)
[ERROR 2015-07-23 05:13:41 main] Repeating errors encountered during run:
[ERROR 2015-07-23 05:13:41 main] Redirecting to /bin/systemctl status  httpd.service
httpd.service - The Apache HTTP Server
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled)
   Active: inactive (dead) since Thu 2015-07-23 04:54:45 EDT; 17min ago
 Main PID: 2086 (code=exited, status=0/SUCCESS)
   Status: "Total requests: 0; Current requests/sec: 0; Current traffic:   0 B/sec"
   CGroup: /system.slice/httpd.service

Jul 23 04:54:40 cloud-qe-20.idmqe.lab.eng.bos.redhat.com pulp[2125]: qpid.messaging:WARNING: recoverable error[attempt 3]: [Errno 111] Connection refused
J

Expected results:
No error should be thrown in logs when upgrade is completed sucessfully as per stdout logs

Additional info:

upgrade completed as per stdout:

[root@cloud-qe-20 ~]# capsule-installer --upgrade --certs-tar cloud-qe-20.idmqe.lab.eng.bos.redhat.com-certs.tar
Upgrading...
Upgrade Step: stop_services...
Upgrade Step: start_mongo...
Upgrade Step: start_httpd...
Upgrade Step: migrate_pulp...
Upgrade Step: Running installer...
Installing             Done                                               [100%] [..................................................................]
  The full log is at /var/log/capsule-installer/capsule-installer.log
Upgrade Step: restart_services...
Katello upgrade completed!
Comment 1 Sachin Ghai 2015-07-23 09:31:50 UTC 
Created attachment 1055263 [details]
katello-service status shows all services are 'active' without any error.
Comment 2 Sachin Ghai 2015-07-23 09:34:18 UTC 
Created attachment 1055265 [details]
foreman-debug from capsule
Comment 4 Sachin Ghai 2015-07-23 09:38:19 UTC 
Created attachment 1055267 [details]
foreman-debug logs from sat server
Comment 5 Sachin Ghai 2015-07-30 07:05:05 UTC 
This issue is also reproducible on satellite server while upgrading sat6.0.8 to sat6.1.1 GA snap15.  Upgrade completed successfully. However following error thrown in installer log on upgrade completion:

---
==> /var/log/katello-installer/katello-installer.log <==
[DEBUG 2015-07-30 02:52:32 main] Updating Expire Metadata for Custom Content

[ INFO 2015-07-30 02:52:32 main] Katello upgrade completed!
[DEBUG 2015-07-30 02:52:32 main] Hook /usr/share/katello-installer/hooks/post/30-upgrade.rb returned [<Logging::Logger:0x599750 name="main">, <Logging::Logger:0x5be49c name="fatal">]
[ INFO 2015-07-30 02:52:32 main] All hooks in group post finished
[DEBUG 2015-07-30 02:52:32 main] Exit with status code: 2 (signal was 2)
[ERROR 2015-07-30 02:52:32 main] Repeating errors encountered during run:
[ERROR 2015-07-30 02:52:32 main] Redirecting to /bin/systemctl status  httpd.service
httpd.service - The Apache HTTP Server
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled)
   Active: inactive (dead) since Thu 2015-07-30 02:31:44 EDT; 11min ago
 Main PID: 12753 (code=exited, status=0/SUCCESS)
   Status: "Total requests: 0; Current requests/sec: 0; Current traffic:   0 B/sec"
   CGroup: /system.slice/httpd.service
---
Comment 6 Sachin Ghai 2015-07-30 07:06:03 UTC 
Updating the defect summary since this issue is reproducible on both Satellite and capsule and on rhel71.
Comment 7 Sachin Ghai 2015-07-31 10:01:03 UTC 
Its not a rhel71 specific issue. I see similar error on rhel66 while upgrading from 6.0.8 to 6.1.1 snap15 c3.

==> /var/log/katello-installer/katello-installer.log <==
[DEBUG 2015-07-31 05:58:35 main] Updating Expire Metadata for Custom Content

[ INFO 2015-07-31 05:58:35 main] Katello upgrade completed!
[DEBUG 2015-07-31 05:58:35 main] Hook /usr/share/katello-installer/hooks/post/30-upgrade.rb returned [<Logging::Logger:0x3fea5e411530 name="main">, <Logging::Logger:0x3fea5e38abac name="fatal">]
[ INFO 2015-07-31 05:58:35 main] All hooks in group post finished
[DEBUG 2015-07-31 05:58:35 main] Exit with status code: 2 (signal was 2)
[ERROR 2015-07-31 05:58:35 main] Repeating errors encountered during run:
[ERROR 2015-07-31 05:58:35 main] <NilClass> nil
[ERROR 2015-07-31 05:58:35 main] httpd is stopped

[DEBUG 2015-07-31 05:58:35 main] Cleaning /etc/katello-installer/d20150731-13508-1sld1w3
[DEBUG 2015-07-31 05:58:35 main] Cleaning /tmp/default_values.yaml
Comment 8 Stephen Benjamin 2015-07-31 11:17:12 UTC 
The source of this problem is `service httpd status` and `systemctl httpd status` will return an error code if the service is stopped.

That's expected, because we're checking if its stopped to know if we need to start it.  

Although we shouldn't log it like that, it's misleading to say its an error.
Comment 11 Bryan Kearney 2016-08-04 20:17:19 UTC 
Moving 6.2 bugs out to sat-backlog.
Comment 12 Bryan Kearney 2017-01-11 19:44:04 UTC 
This is an older bug which I do not envision being fixed in the near term. I am closing this out. If you believe doing so is an issue, please feel free to re-open and provide additional business information. Thank you.
Note You need to log in before you can comment on or make changes to this bug.