Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1394273

Summary: [cli] connection interrupted by pcsd restart results in a traceback
Product: Red Hat Enterprise Linux 6 Reporter: Radek Steiger <rsteiger>
Component: pcsAssignee: Ondrej Mular <omular>
Status: CLOSED ERRATA QA Contact: cluster-qe <cluster-qe>
Severity: unspecified Docs Contact:
Priority: high    
Version: 6.8CC: cfeist, cluster-maint, idevat, omular, tojeline
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: pcs-0.9.155-2.el6 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-03-21 11:04:45 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Radek Steiger 2016-11-11 14:34:18 UTC 
> Description of problem:

An automatic pcsd daemon restart scheduled within the cluster setup execution causes any subsequent connection from pcs cli to produce a traceback due to an interruption. While timing is very much involved, it should be enough as a reproducer to just run cluster setup followed by a cluster start.


> Version-Release number of selected component (if applicable):

pcs-0.9.155-1.el6.x86_64


> Additional info:

[root@virt-011 ~]# pcs cluster setup --name STSRHTS30264 virt-011 virt-012 virt-038 && pcs cluster start --all --wait
Destroying cluster on nodes: virt-011, virt-012, virt-038...
virt-011: Stopping Cluster (pacemaker)...
virt-038: Stopping Cluster (pacemaker)...
virt-012: Stopping Cluster (pacemaker)...
virt-038: Successfully destroyed cluster
virt-012: Successfully destroyed cluster
virt-011: Successfully destroyed cluster

Sending cluster config files to the nodes...
virt-011: Updated cluster.conf...
virt-012: Updated cluster.conf...
virt-038: Updated cluster.conf...

Synchronizing pcsd certificates on nodes virt-011, virt-012, virt-038...
virt-011: Success
virt-012: Success
virt-038: Success

Restarting pcsd on the nodes in order to reload the certificates...
virt-011: Success
virt-012: Success
virt-038: Success
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib64/python2.6/threading.py", line 532, in __bootstrap_inner
    self.run()
  File "/usr/lib64/python2.6/threading.py", line 484, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/lib/python2.6/site-packages/pcs/utils.py", line 1112, in worker
    returncode, output = action(node, *args, **kwargs)
  File "/usr/lib/python2.6/site-packages/pcs/utils.py", line 261, in startCluster
    return sendHTTPRequest(node, 'remote/cluster_start', None, False, not quiet)
  File "/usr/lib/python2.6/site-packages/pcs/utils.py", line 410, in sendHTTPRequest
    result = opener.open(url,data)
  File "/usr/lib64/python2.6/urllib2.py", line 391, in open
    response = self._open(req, data)
  File "/usr/lib64/python2.6/urllib2.py", line 409, in _open
    '_open', req)
  File "/usr/lib64/python2.6/urllib2.py", line 369, in _call_chain
    result = func(*args)
  File "/usr/lib64/python2.6/urllib2.py", line 1198, in https_open
    return self.do_open(httplib.HTTPSConnection, req)
  File "/usr/lib64/python2.6/urllib2.py", line 1163, in do_open
    r = h.getresponse()
  File "/usr/lib64/python2.6/httplib.py", line 1049, in getresponse
    response.begin()
  File "/usr/lib64/python2.6/httplib.py", line 433, in begin
    version, status, reason = self._read_status()
  File "/usr/lib64/python2.6/httplib.py", line 397, in _read_status
    raise BadStatusLine(line)
BadStatusLine

Exception in thread Thread-3:
Traceback (most recent call last):
  File "/usr/lib64/python2.6/threading.py", line 532, in __bootstrap_inner
    self.run()
  File "/usr/lib64/python2.6/threading.py", line 484, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/lib/python2.6/site-packages/pcs/utils.py", line 1112, in worker
    returncode, output = action(node, *args, **kwargs)
  File "/usr/lib/python2.6/site-packages/pcs/utils.py", line 261, in startCluster
    return sendHTTPRequest(node, 'remote/cluster_start', None, False, not quiet)
  File "/usr/lib/python2.6/site-packages/pcs/utils.py", line 410, in sendHTTPRequest
    result = opener.open(url,data)
  File "/usr/lib64/python2.6/urllib2.py", line 391, in open
    response = self._open(req, data)
  File "/usr/lib64/python2.6/urllib2.py", line 409, in _open
    '_open', req)
  File "/usr/lib64/python2.6/urllib2.py", line 369, in _call_chain
    result = func(*args)
  File "/usr/lib64/python2.6/urllib2.py", line 1198, in https_open
    return self.do_open(httplib.HTTPSConnection, req)
  File "/usr/lib64/python2.6/urllib2.py", line 1163, in do_open
    r = h.getresponse()
  File "/usr/lib64/python2.6/httplib.py", line 1049, in getresponse
    response.begin()
  File "/usr/lib64/python2.6/httplib.py", line 433, in begin
    version, status, reason = self._read_status()
  File "/usr/lib64/python2.6/httplib.py", line 397, in _read_status
    raise BadStatusLine(line)
BadStatusLine

Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/lib64/python2.6/threading.py", line 532, in __bootstrap_inner
    self.run()
  File "/usr/lib64/python2.6/threading.py", line 484, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/lib/python2.6/site-packages/pcs/utils.py", line 1112, in worker
    returncode, output = action(node, *args, **kwargs)
  File "/usr/lib/python2.6/site-packages/pcs/utils.py", line 261, in startCluster
    return sendHTTPRequest(node, 'remote/cluster_start', None, False, not quiet)
  File "/usr/lib/python2.6/site-packages/pcs/utils.py", line 410, in sendHTTPRequest
    result = opener.open(url,data)
  File "/usr/lib64/python2.6/urllib2.py", line 391, in open
    response = self._open(req, data)
  File "/usr/lib64/python2.6/urllib2.py", line 409, in _open
    '_open', req)
  File "/usr/lib64/python2.6/urllib2.py", line 369, in _call_chain
    result = func(*args)
  File "/usr/lib64/python2.6/urllib2.py", line 1198, in https_open
    return self.do_open(httplib.HTTPSConnection, req)
  File "/usr/lib64/python2.6/urllib2.py", line 1163, in do_open
    r = h.getresponse()
  File "/usr/lib64/python2.6/httplib.py", line 1049, in getresponse
    response.begin()
  File "/usr/lib64/python2.6/httplib.py", line 433, in begin
    version, status, reason = self._read_status()
  File "/usr/lib64/python2.6/httplib.py", line 397, in _read_status
    raise BadStatusLine(line)
BadStatusLine

Waiting for node(s) to start...
virt-038: Started
virt-012: Started
virt-011: Started
Comment 3 Ondrej Mular 2016-11-16 16:35:24 UTC 
upstream fix:
https://github.com/ClusterLabs/pcs/commit/151fa853c2b56e71987b3cb4d8114c3ed6be

TEST:
[root@rhel6-node1 ~]# pcs cluster setup --name rh6 rh6-{1,2,3} && pcs cluster start --all --wait
Destroying cluster on nodes: rh6-1, rh6-2, rh6-3...
rh6-3: Stopping Cluster (pacemaker)...
rh6-1: Stopping Cluster (pacemaker)...
rh6-2: Stopping Cluster (pacemaker)...
rh6-1: Successfully destroyed cluster
rh6-2: Successfully destroyed cluster
rh6-3: Successfully destroyed cluster

Sending cluster config files to the nodes...
rh6-1: Updated cluster.conf...
rh6-2: Updated cluster.conf...
rh6-3: Updated cluster.conf...

Synchronizing pcsd certificates on nodes rh6-1, rh6-2, rh6-3...
rh6-1: Success
rh6-3: Success
rh6-2: Success

Restarting pcsd on the nodes in order to reload the certificates...
rh6-1: Success
rh6-3: Success
rh6-2: Success
rh6-2: Starting Cluster...
rh6-3: Unable to connect to rh6-3 (Connection error)
rh6-1: Unable to connect to rh6-1 (Connection error)
Error: unable to start all nodes
rh6-1: Unable to connect to rh6-1 (Connection error)
rh6-3: Unable to connect to rh6-3 (Connection error)
Comment 4 Ivan Devat 2016-11-25 11:04:33 UTC 
Before Fix:

[vm-rhel67-1 ~] $ rpm -q pcs
pcs-0.9.155-1.el6.x86_64

Run it multiple times...

[vm-rhel67-1 ~] $ pcs cluster setup --name=devcluster6 vm-rhel67-1 vm-rhel67-2 vm-rhel67-3 && pcs cluster start --all --wait
Destroying cluster on nodes: vm-rhel67-1, vm-rhel67-2, vm-rhel67-3...
vm-rhel67-1: Stopping Cluster (pacemaker)...
vm-rhel67-3: Stopping Cluster (pacemaker)...
vm-rhel67-2: Stopping Cluster (pacemaker)...
vm-rhel67-1: Successfully destroyed cluster
vm-rhel67-3: Successfully destroyed cluster
vm-rhel67-2: Successfully destroyed cluster

Sending cluster config files to the nodes...
vm-rhel67-1: Updated cluster.conf...
vm-rhel67-2: Updated cluster.conf...
vm-rhel67-3: Updated cluster.conf...

Synchronizing pcsd certificates on nodes vm-rhel67-1, vm-rhel67-2, vm-rhel67-3...
vm-rhel67-1: Success
vm-rhel67-2: Success
vm-rhel67-3: Success

Restarting pcsd on the nodes in order to reload the certificates...
vm-rhel67-1: Success
vm-rhel67-2: Success
vm-rhel67-3: Success
Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/lib64/python2.6/threading.py", line 532, in __bootstrap_inner
    self.run()
  File "/usr/lib64/python2.6/threading.py", line 484, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/lib/python2.6/site-packages/pcs/utils.py", line 1112, in worker
    returncode, output = action(node, *args, **kwargs)
  File "/usr/lib/python2.6/site-packages/pcs/utils.py", line 261, in startCluster
    return sendHTTPRequest(node, 'remote/cluster_start', None, False, not quiet)
  File "/usr/lib/python2.6/site-packages/pcs/utils.py", line 410, in sendHTTPRequest
    result = opener.open(url,data)
  File "/usr/lib64/python2.6/urllib2.py", line 391, in open
    response = self._open(req, data)
  File "/usr/lib64/python2.6/urllib2.py", line 409, in _open
    '_open', req)
  File "/usr/lib64/python2.6/urllib2.py", line 369, in _call_chain
    result = func(*args)
  File "/usr/lib64/python2.6/urllib2.py", line 1198, in https_open
    return self.do_open(httplib.HTTPSConnection, req)
  File "/usr/lib64/python2.6/urllib2.py", line 1163, in do_open
    r = h.getresponse()
  File "/usr/lib64/python2.6/httplib.py", line 1012, in getresponse
    response.begin()
  File "/usr/lib64/python2.6/httplib.py", line 404, in begin
    version, status, reason = self._read_status()
  File "/usr/lib64/python2.6/httplib.py", line 368, in _read_status
    raise BadStatusLine(line)
BadStatusLine

Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib64/python2.6/threading.py", line 532, in __bootstrap_inner
    self.run()
  File "/usr/lib64/python2.6/threading.py", line 484, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/lib/python2.6/site-packages/pcs/utils.py", line 1112, in worker
    returncode, output = action(node, *args, **kwargs)
  File "/usr/lib/python2.6/site-packages/pcs/utils.py", line 261, in startCluster
    return sendHTTPRequest(node, 'remote/cluster_start', None, False, not quiet)
  File "/usr/lib/python2.6/site-packages/pcs/utils.py", line 410, in sendHTTPRequest
    result = opener.open(url,data)
  File "/usr/lib64/python2.6/urllib2.py", line 391, in open
    response = self._open(req, data)
  File "/usr/lib64/python2.6/urllib2.py", line 409, in _open
    '_open', req)
  File "/usr/lib64/python2.6/urllib2.py", line 369, in _call_chain
    result = func(*args)
  File "/usr/lib64/python2.6/urllib2.py", line 1198, in https_open
    return self.do_open(httplib.HTTPSConnection, req)
  File "/usr/lib64/python2.6/urllib2.py", line 1163, in do_open
    r = h.getresponse()
  File "/usr/lib64/python2.6/httplib.py", line 1012, in getresponse
    response.begin()
  File "/usr/lib64/python2.6/httplib.py", line 404, in begin
    version, status, reason = self._read_status()
  File "/usr/lib64/python2.6/httplib.py", line 368, in _read_status
    raise BadStatusLine(line)
BadStatusLine

Exception in thread Thread-3:
Traceback (most recent call last):
  File "/usr/lib64/python2.6/threading.py", line 532, in __bootstrap_inner
    self.run()
  File "/usr/lib64/python2.6/threading.py", line 484, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/lib/python2.6/site-packages/pcs/utils.py", line 1112, in worker
    returncode, output = action(node, *args, **kwargs)
  File "/usr/lib/python2.6/site-packages/pcs/utils.py", line 261, in startCluster
    return sendHTTPRequest(node, 'remote/cluster_start', None, False, not quiet)
  File "/usr/lib/python2.6/site-packages/pcs/utils.py", line 410, in sendHTTPRequest
    result = opener.open(url,data)
  File "/usr/lib64/python2.6/urllib2.py", line 391, in open
    response = self._open(req, data)
  File "/usr/lib64/python2.6/urllib2.py", line 409, in _open
    '_open', req)
  File "/usr/lib64/python2.6/urllib2.py", line 369, in _call_chain
    result = func(*args)
  File "/usr/lib64/python2.6/urllib2.py", line 1198, in https_open
    return self.do_open(httplib.HTTPSConnection, req)
  File "/usr/lib64/python2.6/urllib2.py", line 1163, in do_open
    r = h.getresponse()
  File "/usr/lib64/python2.6/httplib.py", line 1012, in getresponse
    response.begin()
  File "/usr/lib64/python2.6/httplib.py", line 404, in begin
    version, status, reason = self._read_status()
  File "/usr/lib64/python2.6/httplib.py", line 368, in _read_status
    raise BadStatusLine(line)
BadStatusLine

Waiting for node(s) to start...
vm-rhel67-1: Error connecting to vm-rhel67-1 - (HTTP error: 400)
vm-rhel67-3: Started
vm-rhel67-2: Started
Error: unable to verify all nodes have started


After Fix:

[vm-rhel67-1 ~] $ rpm -q pcs
pcs-0.9.155-2.el6.x86_64

Run it multiple times...

[vm-rhel67-1 ~] $ pcs cluster setup --name=devcluster6 vm-rhel67-1 vm-rhel67-2 vm-rhel67-3 && pcs cluster start --all --wait
Destroying cluster on nodes: vm-rhel67-1, vm-rhel67-2, vm-rhel67-3...
vm-rhel67-3: Stopping Cluster (pacemaker)...
vm-rhel67-1: Stopping Cluster (pacemaker)...
vm-rhel67-2: Stopping Cluster (pacemaker)...
vm-rhel67-3: Successfully destroyed cluster
vm-rhel67-1: Successfully destroyed cluster
vm-rhel67-2: Successfully destroyed cluster

Sending cluster config files to the nodes...
vm-rhel67-1: Updated cluster.conf...
vm-rhel67-2: Updated cluster.conf...
vm-rhel67-3: Updated cluster.conf...

Synchronizing pcsd certificates on nodes vm-rhel67-1, vm-rhel67-2, vm-rhel67-3...
vm-rhel67-1: Success
vm-rhel67-2: Success
vm-rhel67-3: Success

Restarting pcsd on the nodes in order to reload the certificates...
vm-rhel67-1: Success
vm-rhel67-2: Success
vm-rhel67-3: Success
vm-rhel67-2: Starting Cluster...
vm-rhel67-1: Starting Cluster...
vm-rhel67-3: Starting Cluster...
Waiting for node(s) to start...
vm-rhel67-2: Started
vm-rhel67-3: Started
vm-rhel67-1: Started
Comment 8 errata-xmlrpc 2017-03-21 11:04:45 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2017-0707.html