Note: This bug is displayed in read-only format because
the product is no longer active in Red Hat Bugzilla.
Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
During a Satellite 6.2 upgrade, services utilized by the Satellite will be stopped. Errors related to the stopped services might be observed in the logs.
For example, an error similar to the following may be observed in /var/log/messages while the upgrade is running:
Jun 13 08:39:02 satellite-server virt-who: 2016-06-13 08:39:02,623 ERROR: Error in communication with subscription manager:
This error may be ignored and does not impact the outcome of the upgrade.
Description of problem:
Getting following virt-who related error at the step: migrate_pulp... while performing upgrade from sat6.1.9 -> sat6.2 GA snap15.2 on rhel7
Jun 13 08:39:02 qe-sat6-upgrade-rhel7 virt-who: 2016-06-13 08:39:02,615 DEBUG: Authenticating with RHSM username admin
Jun 13 08:39:02 qe-sat6-upgrade-rhel7 virt-who: 2016-06-13 08:39:02,623 ERROR: Error in communication with subscription manager:
Jun 13 08:39:02 qe-sat6-upgrade-rhel7 virt-who: Traceback (most recent call last):
Jun 13 08:39:02 qe-sat6-upgrade-rhel7 virt-who: File "/usr/share/virt-who/virtwho.py", line 173, in send
Jun 13 08:39:02 qe-sat6-upgrade-rhel7 virt-who: self._sendGuestAssociation(report)
Jun 13 08:39:02 qe-sat6-upgrade-rhel7 virt-who: File "/usr/share/virt-who/virtwho.py", line 204, in _sendGuestAssociation
Jun 13 08:39:02 qe-sat6-upgrade-rhel7 virt-who: report.config.type)
Jun 13 08:39:02 qe-sat6-upgrade-rhel7 virt-who: File "/usr/share/virt-who/manager/subscriptionmanager/subscriptionmanager.py", line 145, in hypervisorCheckIn
Jun 13 08:39:02 qe-sat6-upgrade-rhel7 virt-who: self._connect(config)
Jun 13 08:39:02 qe-sat6-upgrade-rhel7 virt-who: File "/usr/share/virt-who/manager/subscriptionmanager/subscriptionmanager.py", line 120, in _connect
Jun 13 08:39:02 qe-sat6-upgrade-rhel7 virt-who: if not self.connection.ping()['result']:
Jun 13 08:39:02 qe-sat6-upgrade-rhel7 virt-who: File "/usr/lib64/python2.7/site-packages/rhsm/connection.py", line 839, in ping
Jun 13 08:39:02 qe-sat6-upgrade-rhel7 virt-who: return self.conn.request_get("/status/")
Jun 13 08:39:02 qe-sat6-upgrade-rhel7 virt-who: File "/usr/lib64/python2.7/site-packages/rhsm/connection.py", line 644, in request_get
Jun 13 08:39:02 qe-sat6-upgrade-rhel7 virt-who: return self._request("GET", method)
Jun 13 08:39:02 qe-sat6-upgrade-rhel7 virt-who: File "/usr/lib64/python2.7/site-packages/rhsm/connection.py", line 537, in _request
Jun 13 08:39:02 qe-sat6-upgrade-rhel7 virt-who: conn.request(request_type, handler, body=body, headers=headers)
Jun 13 08:39:02 qe-sat6-upgrade-rhel7 virt-who: File "/usr/lib64/python2.7/httplib.py", line 979, in request
Jun 13 08:39:02 qe-sat6-upgrade-rhel7 virt-who: self._send_request(method, url, body, headers)
Jun 13 08:39:02 qe-sat6-upgrade-rhel7 virt-who: File "/usr/lib64/python2.7/httplib.py", line 1013, in _send_request
Jun 13 08:39:02 qe-sat6-upgrade-rhel7 virt-who: self.endheaders(body)
Jun 13 08:39:02 qe-sat6-upgrade-rhel7 virt-who: File "/usr/lib64/python2.7/httplib.py", line 975, in endheaders
Jun 13 08:39:02 qe-sat6-upgrade-rhel7 virt-who: self._send_output(message_body)
Jun 13 08:39:02 qe-sat6-upgrade-rhel7 virt-who: File "/usr/lib64/python2.7/httplib.py", line 835, in _send_output
Jun 13 08:39:02 qe-sat6-upgrade-rhel7 virt-who: self.send(msg)
Jun 13 08:39:02 qe-sat6-upgrade-rhel7 virt-who: File "/usr/lib64/python2.7/httplib.py", line 797, in send
Jun 13 08:39:02 qe-sat6-upgrade-rhel7 virt-who: self.connect()
Jun 13 08:39:02 qe-sat6-upgrade-rhel7 virt-who: File "/usr/lib64/python2.7/site-packages/M2Crypto/httpslib.py", line 73, in connect
Jun 13 08:39:02 qe-sat6-upgrade-rhel7 virt-who: raise error
Jun 13 08:39:02 qe-sat6-upgrade-rhel7 virt-who: error: [Errno 111] Connection refused
Jun 13 08:39:02 qe-sat6-upgrade-rhel7 python: Error in communication with subscription manager:
Version-Release number of selected component (if applicable):
upgrade from sat6.1.9 -> sat6.2 GA snap15.2 on rhel7
How reproducible:
Steps to Reproduce:
1. set sat6.2 repo
2. yum update -y
3. satellite-installer --scenario satellite --upgrade
Actual results:
Expected results:
no error should be thrown in logs while upgrade.
Additional info:
Above error appears at step; migrate_pulp
[root@qe-sat6-upgrade-rhel7 yum.repos.d]# satellite-installer --scenario satellite --upgrade
Upgrading...
Upgrade Step: stop_services...
Upgrade Step: start_databases...
Upgrade Step: update_http_conf...
Upgrade Step: migrate_pulp...
Looks like virt-who is trying to check in but either the candlepin server is down or the credentials are bad "Jun 13 08:39:02 qe-sat6-upgrade-rhel7 virt-who: error: [Errno 111] Connection refused"
Sachin, from looking over the logs, it looks like tomcat was stopped prior to the error message observed and restarted after the error. This would explain the error, as Barnaby mentions in comment 6.
The following illustrates tomcat being stopped prior to the error:
Jun 13 08:34:20 qe-sat6-upgrade-rhel7 systemd: Stopping Apache Tomcat Web Application Container...
Jun 13 08:34:20 qe-sat6-upgrade-rhel7 server: Java virtual machine used: /usr/lib/jvm/jre/bin/java
Jun 13 08:34:20 qe-sat6-upgrade-rhel7 server: classpath used: /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/commons-daemon.jar
Jun 13 08:34:20 qe-sat6-upgrade-rhel7 server: main class used: org.apache.catalina.startup.Bootstrap
Jun 13 08:34:20 qe-sat6-upgrade-rhel7 server: flags used:
Jun 13 08:34:20 qe-sat6-upgrade-rhel7 server: options used: -Dcatalina.base=/usr/share/tomcat -Dcatalina.home=/usr/share/tomcat -Djava.endorsed.dirs= -Djava.io.tmpdir=/var/cache/tomcat/temp -Djava.util.logging.config.file=/usr/share/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
Jun 13 08:34:20 qe-sat6-upgrade-rhel7 server: arguments used: stop
Jun 13 08:34:21 qe-sat6-upgrade-rhel7 server: Jun 13, 2016 8:34:21 AM org.apache.catalina.core.StandardServer await
Jun 13 08:34:21 qe-sat6-upgrade-rhel7 server: INFO: A valid shutdown command was received via the shutdown port. Stopping the Server instance.
Jun 13 08:34:21 qe-sat6-upgrade-rhel7 server: Jun 13, 2016 8:34:21 AM org.apache.coyote.AbstractProtocol pause
Jun 13 08:34:21 qe-sat6-upgrade-rhel7 server: INFO: Pausing ProtocolHandler ["http-bio-8080"]
Jun 13 08:34:21 qe-sat6-upgrade-rhel7 systemd: Stopped Apache Tomcat Web Application Container.
The following illustrates tomcat being restarted after the error occured:
Jun 13 08:44:36 qe-sat6-upgrade-rhel7 systemd: Started Apache Tomcat Web Application Container.
Jun 13 08:44:36 qe-sat6-upgrade-rhel7 systemd: Starting Apache Tomcat Web Application Container...
...
Based on this, this appears to be expected behavior.
Yes, I tried upgrade from sat6.1.9 -> sat6.2 GA snap16 and realized that this error is being thrown when installer stop services while performing upgrade.
Can't we suppress the errors to warning as installer itself stopping services ? This time other service also being stopped but only this tomcat service is throwing error.
Sachin,
yes, it would be nice to suppress errors from clients (sub-man, virt-who, puppet) attempting to interact with the Satellite during upgrade but it would require quite a bit of tooling and reconfiguration of the logging policy to ensure we got this 100% right.
The goal is to have clean output from the satellite-installer --upgrade command but we can't yet ensure all logging is error free while systems are shutdown.
Eventually we can implement and 'Outage Mode' such that we can deflect incoming calls during restart/upgrades but we don't have this now.
Description of problem: Getting following virt-who related error at the step: migrate_pulp... while performing upgrade from sat6.1.9 -> sat6.2 GA snap15.2 on rhel7 Jun 13 08:39:02 qe-sat6-upgrade-rhel7 virt-who: 2016-06-13 08:39:02,615 DEBUG: Authenticating with RHSM username admin Jun 13 08:39:02 qe-sat6-upgrade-rhel7 virt-who: 2016-06-13 08:39:02,623 ERROR: Error in communication with subscription manager: Jun 13 08:39:02 qe-sat6-upgrade-rhel7 virt-who: Traceback (most recent call last): Jun 13 08:39:02 qe-sat6-upgrade-rhel7 virt-who: File "/usr/share/virt-who/virtwho.py", line 173, in send Jun 13 08:39:02 qe-sat6-upgrade-rhel7 virt-who: self._sendGuestAssociation(report) Jun 13 08:39:02 qe-sat6-upgrade-rhel7 virt-who: File "/usr/share/virt-who/virtwho.py", line 204, in _sendGuestAssociation Jun 13 08:39:02 qe-sat6-upgrade-rhel7 virt-who: report.config.type) Jun 13 08:39:02 qe-sat6-upgrade-rhel7 virt-who: File "/usr/share/virt-who/manager/subscriptionmanager/subscriptionmanager.py", line 145, in hypervisorCheckIn Jun 13 08:39:02 qe-sat6-upgrade-rhel7 virt-who: self._connect(config) Jun 13 08:39:02 qe-sat6-upgrade-rhel7 virt-who: File "/usr/share/virt-who/manager/subscriptionmanager/subscriptionmanager.py", line 120, in _connect Jun 13 08:39:02 qe-sat6-upgrade-rhel7 virt-who: if not self.connection.ping()['result']: Jun 13 08:39:02 qe-sat6-upgrade-rhel7 virt-who: File "/usr/lib64/python2.7/site-packages/rhsm/connection.py", line 839, in ping Jun 13 08:39:02 qe-sat6-upgrade-rhel7 virt-who: return self.conn.request_get("/status/") Jun 13 08:39:02 qe-sat6-upgrade-rhel7 virt-who: File "/usr/lib64/python2.7/site-packages/rhsm/connection.py", line 644, in request_get Jun 13 08:39:02 qe-sat6-upgrade-rhel7 virt-who: return self._request("GET", method) Jun 13 08:39:02 qe-sat6-upgrade-rhel7 virt-who: File "/usr/lib64/python2.7/site-packages/rhsm/connection.py", line 537, in _request Jun 13 08:39:02 qe-sat6-upgrade-rhel7 virt-who: conn.request(request_type, handler, body=body, headers=headers) Jun 13 08:39:02 qe-sat6-upgrade-rhel7 virt-who: File "/usr/lib64/python2.7/httplib.py", line 979, in request Jun 13 08:39:02 qe-sat6-upgrade-rhel7 virt-who: self._send_request(method, url, body, headers) Jun 13 08:39:02 qe-sat6-upgrade-rhel7 virt-who: File "/usr/lib64/python2.7/httplib.py", line 1013, in _send_request Jun 13 08:39:02 qe-sat6-upgrade-rhel7 virt-who: self.endheaders(body) Jun 13 08:39:02 qe-sat6-upgrade-rhel7 virt-who: File "/usr/lib64/python2.7/httplib.py", line 975, in endheaders Jun 13 08:39:02 qe-sat6-upgrade-rhel7 virt-who: self._send_output(message_body) Jun 13 08:39:02 qe-sat6-upgrade-rhel7 virt-who: File "/usr/lib64/python2.7/httplib.py", line 835, in _send_output Jun 13 08:39:02 qe-sat6-upgrade-rhel7 virt-who: self.send(msg) Jun 13 08:39:02 qe-sat6-upgrade-rhel7 virt-who: File "/usr/lib64/python2.7/httplib.py", line 797, in send Jun 13 08:39:02 qe-sat6-upgrade-rhel7 virt-who: self.connect() Jun 13 08:39:02 qe-sat6-upgrade-rhel7 virt-who: File "/usr/lib64/python2.7/site-packages/M2Crypto/httpslib.py", line 73, in connect Jun 13 08:39:02 qe-sat6-upgrade-rhel7 virt-who: raise error Jun 13 08:39:02 qe-sat6-upgrade-rhel7 virt-who: error: [Errno 111] Connection refused Jun 13 08:39:02 qe-sat6-upgrade-rhel7 python: Error in communication with subscription manager: Version-Release number of selected component (if applicable): upgrade from sat6.1.9 -> sat6.2 GA snap15.2 on rhel7 How reproducible: Steps to Reproduce: 1. set sat6.2 repo 2. yum update -y 3. satellite-installer --scenario satellite --upgrade Actual results: Expected results: no error should be thrown in logs while upgrade. Additional info: Above error appears at step; migrate_pulp [root@qe-sat6-upgrade-rhel7 yum.repos.d]# satellite-installer --scenario satellite --upgrade Upgrading... Upgrade Step: stop_services... Upgrade Step: start_databases... Upgrade Step: update_http_conf... Upgrade Step: migrate_pulp...