Bug 1245998

Summary:

After upgrade[sat6.0.8 -> sat6.1.1 ], capsule/katello-installer.log throws error on checking "httpd status", though upgrade is completed successfully

Product:

Red Hat Satellite

Reporter:

Sachin Ghai <sghai>

Component:

Upgrades

Assignee:

satellite6-bugs <satellite6-bugs>

Status:

CLOSED WONTFIX

QA Contact:

Sachin Ghai <sghai>

Severity:

medium

Docs Contact:

Priority:

unspecified

Version:

6.1.0

CC:

bbuckingham, bkearney, stbenjam

Target Milestone:

Unspecified

Keywords:

Triaged

Target Release:

Unused

Hardware:

Unspecified

OS:

Unspecified

Whiteboard:

Fixed In Version:

Doc Type:

Bug Fix

Doc Text:

Story Points:

---

Clone Of:

Environment:

Last Closed:

2017-01-11 19:44:04 UTC

Type:

Bug

Regression:

---

Mount Type:

---

Documentation:

---

CRM:

Verified Versions:

Category:

---

oVirt Team:

---

RHEL 7.3 requirements from Atomic Host:

Cloudforms Team:

---

Target Upstream Version:

Embargoed:

Bug Depends On:

Bug Blocks:

1195450

Attachments:

Description	Flags
katello-service status shows all services are 'active' without any error.	none
foreman-debug from capsule	none
foreman-debug logs from sat server	none

Description Sachin Ghai 2015-07-23 09:30:32 UTC

Description of problem:
I was trying capsule-upgrade from sat6.0.8 to sat6.1.1 GA snap14 on rhel71. Upgrade completed successfully. No error on stdout. However when I checked the capsule-installer.log, it ends with below error:

-----------
[ INFO 2015-07-23 05:13:28 main] Upgrade Step: restart_services...
[DEBUG 2015-07-23 05:13:41 main] Redirecting to /bin/systemctl restart  qpidd.service

Redirecting to /bin/systemctl restart  qdrouterd.service

Redirecting to /bin/systemctl restart  pulp_workers.service

Redirecting to /bin/systemctl restart  foreman-proxy.service

Redirecting to /bin/systemctl restart  pulp_resource_manager.service

Redirecting to /bin/systemctl restart  mongod.service

Redirecting to /bin/systemctl restart  pulp_celerybeat.service

Redirecting to /bin/systemctl restart  httpd.service

Success!

[ INFO 2015-07-23 05:13:41 main] Katello upgrade completed!
[DEBUG 2015-07-23 05:13:41 main] Hook /usr/share/katello-installer/hooks/post/30-upgrade.rb returned [<Logging::Logger:0x1ab5dc8 name="main">, <Logging::Logger:0x1aba0e4 name="fatal">]
[ INFO 2015-07-23 05:13:41 main] All hooks in group post finished
[DEBUG 2015-07-23 05:13:41 main] Exit with status code: 2 (signal was 2)
[ERROR 2015-07-23 05:13:41 main] Repeating errors encountered during run:
[ERROR 2015-07-23 05:13:41 main] Redirecting to /bin/systemctl status  httpd.service
httpd.service - The Apache HTTP Server
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled)
   Active: inactive (dead) since Thu 2015-07-23 04:54:45 EDT; 17min ago
 Main PID: 2086 (code=exited, status=0/SUCCESS)
   Status: "Total requests: 0; Current requests/sec: 0; Current traffic:   0 B/sec"
   CGroup: /system.slice/httpd.service

Jul 23 04:54:40 cloud-qe-20.idmqe.lab.eng.bos.redhat.com pulp[2125]: qpid.messaging:WARNING: recoverable error[attempt 3]: [Errno 111] Connection refused
Jul 23 04:54:40 cloud-qe-20.idmqe.lab.eng.bos.redhat.com pulp[2125]: qpid.messaging:WARNING: sleeping 4 seconds
Jul 23 04:54:40 cloud-qe-20.idmqe.lab.eng.bos.redhat.com pulp[2125]: qpid.messaging:WARNING: trying: cloud-qe-20.idmqe.lab.eng.bos.redhat.com:5671
Jul 23 04:54:40 cloud-qe-20.idmqe.lab.eng.bos.redhat.com pulp[2125]: qpid.messaging:WARNING: recoverable error[attempt 3]: [Errno 111] Connection refused
Jul 23 04:54:40 cloud-qe-20.idmqe.lab.eng.bos.redhat.com pulp[2125]: qpid.messaging:WARNING: sleeping 4 seconds
Jul 23 04:54:41 cloud-qe-20.idmqe.lab.eng.bos.redhat.com systemd[1]: Stopping The Apache HTTP Server...
Jul 23 04:54:45 cloud-qe-20.idmqe.lab.eng.bos.redhat.com pulp[2125]: qpid.messaging:WARNING: trying: cloud-qe-20.idmqe.lab.eng.bos.redhat.com:5671
Jul 23 04:54:45 cloud-qe-20.idmqe.lab.eng.bos.redhat.com pulp[2125]: qpid.messaging:WARNING: recoverable error[attempt 4]: [Errno 111] Connection refused
Jul 23 04:54:45 cloud-qe-20.idmqe.lab.eng.bos.redhat.com pulp[2125]: qpid.messaging:WARNING: sleeping 8 seconds
Jul 23 04:54:45 cloud-qe-20.idmqe.lab.eng.bos.redhat.com systemd[1]: Stopped The Apache HTTP Server.
-----------------


On manually, however httpd status is 'ok' when checking the httpd status after upgrade.


Version-Release number of selected component (if applicable):
sat6.1.1 GA snap14

How reproducible:


Steps to Reproduce:
1. capsule-installer --upgrade --certs-tar <tarball>
2.
3.

Actual results:
[ INFO 2015-07-23 05:13:41 main] Katello upgrade completed!
[DEBUG 2015-07-23 05:13:41 main] Hook /usr/share/katello-installer/hooks/post/30-upgrade.rb returned [<Logging::Logger:0x1ab5dc8 name="main">, <Logging::Logger:0x1aba0e4 name="fatal">]
[ INFO 2015-07-23 05:13:41 main] All hooks in group post finished
[DEBUG 2015-07-23 05:13:41 main] Exit with status code: 2 (signal was 2)
[ERROR 2015-07-23 05:13:41 main] Repeating errors encountered during run:
[ERROR 2015-07-23 05:13:41 main] Redirecting to /bin/systemctl status  httpd.service
httpd.service - The Apache HTTP Server
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled)
   Active: inactive (dead) since Thu 2015-07-23 04:54:45 EDT; 17min ago
 Main PID: 2086 (code=exited, status=0/SUCCESS)
   Status: "Total requests: 0; Current requests/sec: 0; Current traffic:   0 B/sec"
   CGroup: /system.slice/httpd.service

Jul 23 04:54:40 cloud-qe-20.idmqe.lab.eng.bos.redhat.com pulp[2125]: qpid.messaging:WARNING: recoverable error[attempt 3]: [Errno 111] Connection refused
J

Expected results:
No error should be thrown in logs when upgrade is completed sucessfully as per stdout logs

Additional info:

upgrade completed as per stdout:

[root@cloud-qe-20 ~]# capsule-installer --upgrade --certs-tar cloud-qe-20.idmqe.lab.eng.bos.redhat.com-certs.tar
Upgrading...
Upgrade Step: stop_services...
Upgrade Step: start_mongo...
Upgrade Step: start_httpd...
Upgrade Step: migrate_pulp...
Upgrade Step: Running installer...
Installing             Done                                               [100%] [..................................................................]
  The full log is at /var/log/capsule-installer/capsule-installer.log
Upgrade Step: restart_services...
Katello upgrade completed!

Comment 1 Sachin Ghai 2015-07-23 09:31:50 UTC

Created attachment 1055263 [details]
katello-service status shows all services are 'active' without any error.

Comment 2 Sachin Ghai 2015-07-23 09:34:18 UTC

Created attachment 1055265 [details]
foreman-debug from capsule

Comment 4 Sachin Ghai 2015-07-23 09:38:19 UTC

Created attachment 1055267 [details]
foreman-debug logs from sat server

Comment 5 Sachin Ghai 2015-07-30 07:05:05 UTC

This issue is also reproducible on satellite server while upgrading sat6.0.8 to sat6.1.1 GA snap15.  Upgrade completed successfully. However following error thrown in installer log on upgrade completion:

---
==> /var/log/katello-installer/katello-installer.log <==
[DEBUG 2015-07-30 02:52:32 main] Updating Expire Metadata for Custom Content

[ INFO 2015-07-30 02:52:32 main] Katello upgrade completed!
[DEBUG 2015-07-30 02:52:32 main] Hook /usr/share/katello-installer/hooks/post/30-upgrade.rb returned [<Logging::Logger:0x599750 name="main">, <Logging::Logger:0x5be49c name="fatal">]
[ INFO 2015-07-30 02:52:32 main] All hooks in group post finished
[DEBUG 2015-07-30 02:52:32 main] Exit with status code: 2 (signal was 2)
[ERROR 2015-07-30 02:52:32 main] Repeating errors encountered during run:
[ERROR 2015-07-30 02:52:32 main] Redirecting to /bin/systemctl status  httpd.service
httpd.service - The Apache HTTP Server
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled)
   Active: inactive (dead) since Thu 2015-07-30 02:31:44 EDT; 11min ago
 Main PID: 12753 (code=exited, status=0/SUCCESS)
   Status: "Total requests: 0; Current requests/sec: 0; Current traffic:   0 B/sec"
   CGroup: /system.slice/httpd.service
---

Comment 6 Sachin Ghai 2015-07-30 07:06:03 UTC

Updating the defect summary since this issue is reproducible on both Satellite and capsule and on rhel71.

Comment 7 Sachin Ghai 2015-07-31 10:01:03 UTC

Its not a rhel71 specific issue. I see similar error on rhel66 while upgrading from 6.0.8 to 6.1.1 snap15 c3.

==> /var/log/katello-installer/katello-installer.log <==
[DEBUG 2015-07-31 05:58:35 main] Updating Expire Metadata for Custom Content

[ INFO 2015-07-31 05:58:35 main] Katello upgrade completed!
[DEBUG 2015-07-31 05:58:35 main] Hook /usr/share/katello-installer/hooks/post/30-upgrade.rb returned [<Logging::Logger:0x3fea5e411530 name="main">, <Logging::Logger:0x3fea5e38abac name="fatal">]
[ INFO 2015-07-31 05:58:35 main] All hooks in group post finished
[DEBUG 2015-07-31 05:58:35 main] Exit with status code: 2 (signal was 2)
[ERROR 2015-07-31 05:58:35 main] Repeating errors encountered during run:
[ERROR 2015-07-31 05:58:35 main] <NilClass> nil
[ERROR 2015-07-31 05:58:35 main] httpd is stopped

[DEBUG 2015-07-31 05:58:35 main] Cleaning /etc/katello-installer/d20150731-13508-1sld1w3
[DEBUG 2015-07-31 05:58:35 main] Cleaning /tmp/default_values.yaml

Comment 8 Stephen Benjamin 2015-07-31 11:17:12 UTC

The source of this problem is `service httpd status` and `systemctl httpd status` will return an error code if the service is stopped.

That's expected, because we're checking if its stopped to know if we need to start it.  

Although we shouldn't log it like that, it's misleading to say its an error.

Comment 11 Bryan Kearney 2016-08-04 20:17:19 UTC

Moving 6.2 bugs out to sat-backlog.

Comment 12 Bryan Kearney 2017-01-11 19:44:04 UTC

This is an older bug which I do not envision being fixed in the near term. I am closing this out. If you believe doing so is an issue, please feel free to re-open and provide additional business information. Thank you.