RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1394273 - [cli] connection interrupted by pcsd restart results in a traceback
Summary: [cli] connection interrupted by pcsd restart results in a traceback
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: pcs
Version: 6.8
Hardware: Unspecified
OS: Unspecified
high
unspecified
Target Milestone: rc
: ---
Assignee: Ondrej Mular
QA Contact: cluster-qe@redhat.com
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-11-11 14:34 UTC by Radek Steiger
Modified: 2017-03-21 11:04 UTC (History)
5 users (show)

Fixed In Version: pcs-0.9.155-2.el6
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-03-21 11:04:45 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Bugzilla 1284404 0 medium CLOSED make restarting pcsd a synchronous operation 2021-02-22 00:41:40 UTC
Red Hat Product Errata RHBA-2017:0707 0 normal SHIPPED_LIVE pcs bug fix update 2017-03-21 12:40:33 UTC

Internal Links: 1284404

Description Radek Steiger 2016-11-11 14:34:18 UTC
> Description of problem:

An automatic pcsd daemon restart scheduled within the cluster setup execution causes any subsequent connection from pcs cli to produce a traceback due to an interruption. While timing is very much involved, it should be enough as a reproducer to just run cluster setup followed by a cluster start.


> Version-Release number of selected component (if applicable):

pcs-0.9.155-1.el6.x86_64


> Additional info:

[root@virt-011 ~]# pcs cluster setup --name STSRHTS30264 virt-011 virt-012 virt-038 && pcs cluster start --all --wait
Destroying cluster on nodes: virt-011, virt-012, virt-038...
virt-011: Stopping Cluster (pacemaker)...
virt-038: Stopping Cluster (pacemaker)...
virt-012: Stopping Cluster (pacemaker)...
virt-038: Successfully destroyed cluster
virt-012: Successfully destroyed cluster
virt-011: Successfully destroyed cluster

Sending cluster config files to the nodes...
virt-011: Updated cluster.conf...
virt-012: Updated cluster.conf...
virt-038: Updated cluster.conf...

Synchronizing pcsd certificates on nodes virt-011, virt-012, virt-038...
virt-011: Success
virt-012: Success
virt-038: Success

Restarting pcsd on the nodes in order to reload the certificates...
virt-011: Success
virt-012: Success
virt-038: Success
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib64/python2.6/threading.py", line 532, in __bootstrap_inner
    self.run()
  File "/usr/lib64/python2.6/threading.py", line 484, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/lib/python2.6/site-packages/pcs/utils.py", line 1112, in worker
    returncode, output = action(node, *args, **kwargs)
  File "/usr/lib/python2.6/site-packages/pcs/utils.py", line 261, in startCluster
    return sendHTTPRequest(node, 'remote/cluster_start', None, False, not quiet)
  File "/usr/lib/python2.6/site-packages/pcs/utils.py", line 410, in sendHTTPRequest
    result = opener.open(url,data)
  File "/usr/lib64/python2.6/urllib2.py", line 391, in open
    response = self._open(req, data)
  File "/usr/lib64/python2.6/urllib2.py", line 409, in _open
    '_open', req)
  File "/usr/lib64/python2.6/urllib2.py", line 369, in _call_chain
    result = func(*args)
  File "/usr/lib64/python2.6/urllib2.py", line 1198, in https_open
    return self.do_open(httplib.HTTPSConnection, req)
  File "/usr/lib64/python2.6/urllib2.py", line 1163, in do_open
    r = h.getresponse()
  File "/usr/lib64/python2.6/httplib.py", line 1049, in getresponse
    response.begin()
  File "/usr/lib64/python2.6/httplib.py", line 433, in begin
    version, status, reason = self._read_status()
  File "/usr/lib64/python2.6/httplib.py", line 397, in _read_status
    raise BadStatusLine(line)
BadStatusLine

Exception in thread Thread-3:
Traceback (most recent call last):
  File "/usr/lib64/python2.6/threading.py", line 532, in __bootstrap_inner
    self.run()
  File "/usr/lib64/python2.6/threading.py", line 484, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/lib/python2.6/site-packages/pcs/utils.py", line 1112, in worker
    returncode, output = action(node, *args, **kwargs)
  File "/usr/lib/python2.6/site-packages/pcs/utils.py", line 261, in startCluster
    return sendHTTPRequest(node, 'remote/cluster_start', None, False, not quiet)
  File "/usr/lib/python2.6/site-packages/pcs/utils.py", line 410, in sendHTTPRequest
    result = opener.open(url,data)
  File "/usr/lib64/python2.6/urllib2.py", line 391, in open
    response = self._open(req, data)
  File "/usr/lib64/python2.6/urllib2.py", line 409, in _open
    '_open', req)
  File "/usr/lib64/python2.6/urllib2.py", line 369, in _call_chain
    result = func(*args)
  File "/usr/lib64/python2.6/urllib2.py", line 1198, in https_open
    return self.do_open(httplib.HTTPSConnection, req)
  File "/usr/lib64/python2.6/urllib2.py", line 1163, in do_open
    r = h.getresponse()
  File "/usr/lib64/python2.6/httplib.py", line 1049, in getresponse
    response.begin()
  File "/usr/lib64/python2.6/httplib.py", line 433, in begin
    version, status, reason = self._read_status()
  File "/usr/lib64/python2.6/httplib.py", line 397, in _read_status
    raise BadStatusLine(line)
BadStatusLine

Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/lib64/python2.6/threading.py", line 532, in __bootstrap_inner
    self.run()
  File "/usr/lib64/python2.6/threading.py", line 484, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/lib/python2.6/site-packages/pcs/utils.py", line 1112, in worker
    returncode, output = action(node, *args, **kwargs)
  File "/usr/lib/python2.6/site-packages/pcs/utils.py", line 261, in startCluster
    return sendHTTPRequest(node, 'remote/cluster_start', None, False, not quiet)
  File "/usr/lib/python2.6/site-packages/pcs/utils.py", line 410, in sendHTTPRequest
    result = opener.open(url,data)
  File "/usr/lib64/python2.6/urllib2.py", line 391, in open
    response = self._open(req, data)
  File "/usr/lib64/python2.6/urllib2.py", line 409, in _open
    '_open', req)
  File "/usr/lib64/python2.6/urllib2.py", line 369, in _call_chain
    result = func(*args)
  File "/usr/lib64/python2.6/urllib2.py", line 1198, in https_open
    return self.do_open(httplib.HTTPSConnection, req)
  File "/usr/lib64/python2.6/urllib2.py", line 1163, in do_open
    r = h.getresponse()
  File "/usr/lib64/python2.6/httplib.py", line 1049, in getresponse
    response.begin()
  File "/usr/lib64/python2.6/httplib.py", line 433, in begin
    version, status, reason = self._read_status()
  File "/usr/lib64/python2.6/httplib.py", line 397, in _read_status
    raise BadStatusLine(line)
BadStatusLine

Waiting for node(s) to start...
virt-038: Started
virt-012: Started
virt-011: Started

Comment 3 Ondrej Mular 2016-11-16 16:35:24 UTC
upstream fix:
https://github.com/ClusterLabs/pcs/commit/151fa853c2b56e71987b3cb4d8114c3ed6be

TEST:
[root@rhel6-node1 ~]# pcs cluster setup --name rh6 rh6-{1,2,3} && pcs cluster start --all --wait
Destroying cluster on nodes: rh6-1, rh6-2, rh6-3...
rh6-3: Stopping Cluster (pacemaker)...
rh6-1: Stopping Cluster (pacemaker)...
rh6-2: Stopping Cluster (pacemaker)...
rh6-1: Successfully destroyed cluster
rh6-2: Successfully destroyed cluster
rh6-3: Successfully destroyed cluster

Sending cluster config files to the nodes...
rh6-1: Updated cluster.conf...
rh6-2: Updated cluster.conf...
rh6-3: Updated cluster.conf...

Synchronizing pcsd certificates on nodes rh6-1, rh6-2, rh6-3...
rh6-1: Success
rh6-3: Success
rh6-2: Success

Restarting pcsd on the nodes in order to reload the certificates...
rh6-1: Success
rh6-3: Success
rh6-2: Success
rh6-2: Starting Cluster...
rh6-3: Unable to connect to rh6-3 (Connection error)
rh6-1: Unable to connect to rh6-1 (Connection error)
Error: unable to start all nodes
rh6-1: Unable to connect to rh6-1 (Connection error)
rh6-3: Unable to connect to rh6-3 (Connection error)

Comment 4 Ivan Devat 2016-11-25 11:04:33 UTC
Before Fix:

[vm-rhel67-1 ~] $ rpm -q pcs
pcs-0.9.155-1.el6.x86_64

Run it multiple times...

[vm-rhel67-1 ~] $ pcs cluster setup --name=devcluster6 vm-rhel67-1 vm-rhel67-2 vm-rhel67-3 && pcs cluster start --all --wait
Destroying cluster on nodes: vm-rhel67-1, vm-rhel67-2, vm-rhel67-3...
vm-rhel67-1: Stopping Cluster (pacemaker)...
vm-rhel67-3: Stopping Cluster (pacemaker)...
vm-rhel67-2: Stopping Cluster (pacemaker)...
vm-rhel67-1: Successfully destroyed cluster
vm-rhel67-3: Successfully destroyed cluster
vm-rhel67-2: Successfully destroyed cluster

Sending cluster config files to the nodes...
vm-rhel67-1: Updated cluster.conf...
vm-rhel67-2: Updated cluster.conf...
vm-rhel67-3: Updated cluster.conf...

Synchronizing pcsd certificates on nodes vm-rhel67-1, vm-rhel67-2, vm-rhel67-3...
vm-rhel67-1: Success
vm-rhel67-2: Success
vm-rhel67-3: Success

Restarting pcsd on the nodes in order to reload the certificates...
vm-rhel67-1: Success
vm-rhel67-2: Success
vm-rhel67-3: Success
Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/lib64/python2.6/threading.py", line 532, in __bootstrap_inner
    self.run()
  File "/usr/lib64/python2.6/threading.py", line 484, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/lib/python2.6/site-packages/pcs/utils.py", line 1112, in worker
    returncode, output = action(node, *args, **kwargs)
  File "/usr/lib/python2.6/site-packages/pcs/utils.py", line 261, in startCluster
    return sendHTTPRequest(node, 'remote/cluster_start', None, False, not quiet)
  File "/usr/lib/python2.6/site-packages/pcs/utils.py", line 410, in sendHTTPRequest
    result = opener.open(url,data)
  File "/usr/lib64/python2.6/urllib2.py", line 391, in open
    response = self._open(req, data)
  File "/usr/lib64/python2.6/urllib2.py", line 409, in _open
    '_open', req)
  File "/usr/lib64/python2.6/urllib2.py", line 369, in _call_chain
    result = func(*args)
  File "/usr/lib64/python2.6/urllib2.py", line 1198, in https_open
    return self.do_open(httplib.HTTPSConnection, req)
  File "/usr/lib64/python2.6/urllib2.py", line 1163, in do_open
    r = h.getresponse()
  File "/usr/lib64/python2.6/httplib.py", line 1012, in getresponse
    response.begin()
  File "/usr/lib64/python2.6/httplib.py", line 404, in begin
    version, status, reason = self._read_status()
  File "/usr/lib64/python2.6/httplib.py", line 368, in _read_status
    raise BadStatusLine(line)
BadStatusLine

Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib64/python2.6/threading.py", line 532, in __bootstrap_inner
    self.run()
  File "/usr/lib64/python2.6/threading.py", line 484, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/lib/python2.6/site-packages/pcs/utils.py", line 1112, in worker
    returncode, output = action(node, *args, **kwargs)
  File "/usr/lib/python2.6/site-packages/pcs/utils.py", line 261, in startCluster
    return sendHTTPRequest(node, 'remote/cluster_start', None, False, not quiet)
  File "/usr/lib/python2.6/site-packages/pcs/utils.py", line 410, in sendHTTPRequest
    result = opener.open(url,data)
  File "/usr/lib64/python2.6/urllib2.py", line 391, in open
    response = self._open(req, data)
  File "/usr/lib64/python2.6/urllib2.py", line 409, in _open
    '_open', req)
  File "/usr/lib64/python2.6/urllib2.py", line 369, in _call_chain
    result = func(*args)
  File "/usr/lib64/python2.6/urllib2.py", line 1198, in https_open
    return self.do_open(httplib.HTTPSConnection, req)
  File "/usr/lib64/python2.6/urllib2.py", line 1163, in do_open
    r = h.getresponse()
  File "/usr/lib64/python2.6/httplib.py", line 1012, in getresponse
    response.begin()
  File "/usr/lib64/python2.6/httplib.py", line 404, in begin
    version, status, reason = self._read_status()
  File "/usr/lib64/python2.6/httplib.py", line 368, in _read_status
    raise BadStatusLine(line)
BadStatusLine

Exception in thread Thread-3:
Traceback (most recent call last):
  File "/usr/lib64/python2.6/threading.py", line 532, in __bootstrap_inner
    self.run()
  File "/usr/lib64/python2.6/threading.py", line 484, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/lib/python2.6/site-packages/pcs/utils.py", line 1112, in worker
    returncode, output = action(node, *args, **kwargs)
  File "/usr/lib/python2.6/site-packages/pcs/utils.py", line 261, in startCluster
    return sendHTTPRequest(node, 'remote/cluster_start', None, False, not quiet)
  File "/usr/lib/python2.6/site-packages/pcs/utils.py", line 410, in sendHTTPRequest
    result = opener.open(url,data)
  File "/usr/lib64/python2.6/urllib2.py", line 391, in open
    response = self._open(req, data)
  File "/usr/lib64/python2.6/urllib2.py", line 409, in _open
    '_open', req)
  File "/usr/lib64/python2.6/urllib2.py", line 369, in _call_chain
    result = func(*args)
  File "/usr/lib64/python2.6/urllib2.py", line 1198, in https_open
    return self.do_open(httplib.HTTPSConnection, req)
  File "/usr/lib64/python2.6/urllib2.py", line 1163, in do_open
    r = h.getresponse()
  File "/usr/lib64/python2.6/httplib.py", line 1012, in getresponse
    response.begin()
  File "/usr/lib64/python2.6/httplib.py", line 404, in begin
    version, status, reason = self._read_status()
  File "/usr/lib64/python2.6/httplib.py", line 368, in _read_status
    raise BadStatusLine(line)
BadStatusLine

Waiting for node(s) to start...
vm-rhel67-1: Error connecting to vm-rhel67-1 - (HTTP error: 400)
vm-rhel67-3: Started
vm-rhel67-2: Started
Error: unable to verify all nodes have started


After Fix:

[vm-rhel67-1 ~] $ rpm -q pcs
pcs-0.9.155-2.el6.x86_64

Run it multiple times...

[vm-rhel67-1 ~] $ pcs cluster setup --name=devcluster6 vm-rhel67-1 vm-rhel67-2 vm-rhel67-3 && pcs cluster start --all --wait
Destroying cluster on nodes: vm-rhel67-1, vm-rhel67-2, vm-rhel67-3...
vm-rhel67-3: Stopping Cluster (pacemaker)...
vm-rhel67-1: Stopping Cluster (pacemaker)...
vm-rhel67-2: Stopping Cluster (pacemaker)...
vm-rhel67-3: Successfully destroyed cluster
vm-rhel67-1: Successfully destroyed cluster
vm-rhel67-2: Successfully destroyed cluster

Sending cluster config files to the nodes...
vm-rhel67-1: Updated cluster.conf...
vm-rhel67-2: Updated cluster.conf...
vm-rhel67-3: Updated cluster.conf...

Synchronizing pcsd certificates on nodes vm-rhel67-1, vm-rhel67-2, vm-rhel67-3...
vm-rhel67-1: Success
vm-rhel67-2: Success
vm-rhel67-3: Success

Restarting pcsd on the nodes in order to reload the certificates...
vm-rhel67-1: Success
vm-rhel67-2: Success
vm-rhel67-3: Success
vm-rhel67-2: Starting Cluster...
vm-rhel67-1: Starting Cluster...
vm-rhel67-3: Starting Cluster...
Waiting for node(s) to start...
vm-rhel67-2: Started
vm-rhel67-3: Started
vm-rhel67-1: Started

Comment 8 errata-xmlrpc 2017-03-21 11:04:45 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2017-0707.html


Note You need to log in before you can comment on or make changes to this bug.