Hide Forgot
> Description of problem: An automatic pcsd daemon restart scheduled within the cluster setup execution causes any subsequent connection from pcs cli to produce a traceback due to an interruption. While timing is very much involved, it should be enough as a reproducer to just run cluster setup followed by a cluster start. > Version-Release number of selected component (if applicable): pcs-0.9.155-1.el6.x86_64 > Additional info: [root@virt-011 ~]# pcs cluster setup --name STSRHTS30264 virt-011 virt-012 virt-038 && pcs cluster start --all --wait Destroying cluster on nodes: virt-011, virt-012, virt-038... virt-011: Stopping Cluster (pacemaker)... virt-038: Stopping Cluster (pacemaker)... virt-012: Stopping Cluster (pacemaker)... virt-038: Successfully destroyed cluster virt-012: Successfully destroyed cluster virt-011: Successfully destroyed cluster Sending cluster config files to the nodes... virt-011: Updated cluster.conf... virt-012: Updated cluster.conf... virt-038: Updated cluster.conf... Synchronizing pcsd certificates on nodes virt-011, virt-012, virt-038... virt-011: Success virt-012: Success virt-038: Success Restarting pcsd on the nodes in order to reload the certificates... virt-011: Success virt-012: Success virt-038: Success Exception in thread Thread-1: Traceback (most recent call last): File "/usr/lib64/python2.6/threading.py", line 532, in __bootstrap_inner self.run() File "/usr/lib64/python2.6/threading.py", line 484, in run self.__target(*self.__args, **self.__kwargs) File "/usr/lib/python2.6/site-packages/pcs/utils.py", line 1112, in worker returncode, output = action(node, *args, **kwargs) File "/usr/lib/python2.6/site-packages/pcs/utils.py", line 261, in startCluster return sendHTTPRequest(node, 'remote/cluster_start', None, False, not quiet) File "/usr/lib/python2.6/site-packages/pcs/utils.py", line 410, in sendHTTPRequest result = opener.open(url,data) File "/usr/lib64/python2.6/urllib2.py", line 391, in open response = self._open(req, data) File "/usr/lib64/python2.6/urllib2.py", line 409, in _open '_open', req) File "/usr/lib64/python2.6/urllib2.py", line 369, in _call_chain result = func(*args) File "/usr/lib64/python2.6/urllib2.py", line 1198, in https_open return self.do_open(httplib.HTTPSConnection, req) File "/usr/lib64/python2.6/urllib2.py", line 1163, in do_open r = h.getresponse() File "/usr/lib64/python2.6/httplib.py", line 1049, in getresponse response.begin() File "/usr/lib64/python2.6/httplib.py", line 433, in begin version, status, reason = self._read_status() File "/usr/lib64/python2.6/httplib.py", line 397, in _read_status raise BadStatusLine(line) BadStatusLine Exception in thread Thread-3: Traceback (most recent call last): File "/usr/lib64/python2.6/threading.py", line 532, in __bootstrap_inner self.run() File "/usr/lib64/python2.6/threading.py", line 484, in run self.__target(*self.__args, **self.__kwargs) File "/usr/lib/python2.6/site-packages/pcs/utils.py", line 1112, in worker returncode, output = action(node, *args, **kwargs) File "/usr/lib/python2.6/site-packages/pcs/utils.py", line 261, in startCluster return sendHTTPRequest(node, 'remote/cluster_start', None, False, not quiet) File "/usr/lib/python2.6/site-packages/pcs/utils.py", line 410, in sendHTTPRequest result = opener.open(url,data) File "/usr/lib64/python2.6/urllib2.py", line 391, in open response = self._open(req, data) File "/usr/lib64/python2.6/urllib2.py", line 409, in _open '_open', req) File "/usr/lib64/python2.6/urllib2.py", line 369, in _call_chain result = func(*args) File "/usr/lib64/python2.6/urllib2.py", line 1198, in https_open return self.do_open(httplib.HTTPSConnection, req) File "/usr/lib64/python2.6/urllib2.py", line 1163, in do_open r = h.getresponse() File "/usr/lib64/python2.6/httplib.py", line 1049, in getresponse response.begin() File "/usr/lib64/python2.6/httplib.py", line 433, in begin version, status, reason = self._read_status() File "/usr/lib64/python2.6/httplib.py", line 397, in _read_status raise BadStatusLine(line) BadStatusLine Exception in thread Thread-2: Traceback (most recent call last): File "/usr/lib64/python2.6/threading.py", line 532, in __bootstrap_inner self.run() File "/usr/lib64/python2.6/threading.py", line 484, in run self.__target(*self.__args, **self.__kwargs) File "/usr/lib/python2.6/site-packages/pcs/utils.py", line 1112, in worker returncode, output = action(node, *args, **kwargs) File "/usr/lib/python2.6/site-packages/pcs/utils.py", line 261, in startCluster return sendHTTPRequest(node, 'remote/cluster_start', None, False, not quiet) File "/usr/lib/python2.6/site-packages/pcs/utils.py", line 410, in sendHTTPRequest result = opener.open(url,data) File "/usr/lib64/python2.6/urllib2.py", line 391, in open response = self._open(req, data) File "/usr/lib64/python2.6/urllib2.py", line 409, in _open '_open', req) File "/usr/lib64/python2.6/urllib2.py", line 369, in _call_chain result = func(*args) File "/usr/lib64/python2.6/urllib2.py", line 1198, in https_open return self.do_open(httplib.HTTPSConnection, req) File "/usr/lib64/python2.6/urllib2.py", line 1163, in do_open r = h.getresponse() File "/usr/lib64/python2.6/httplib.py", line 1049, in getresponse response.begin() File "/usr/lib64/python2.6/httplib.py", line 433, in begin version, status, reason = self._read_status() File "/usr/lib64/python2.6/httplib.py", line 397, in _read_status raise BadStatusLine(line) BadStatusLine Waiting for node(s) to start... virt-038: Started virt-012: Started virt-011: Started
upstream fix: https://github.com/ClusterLabs/pcs/commit/151fa853c2b56e71987b3cb4d8114c3ed6be TEST: [root@rhel6-node1 ~]# pcs cluster setup --name rh6 rh6-{1,2,3} && pcs cluster start --all --wait Destroying cluster on nodes: rh6-1, rh6-2, rh6-3... rh6-3: Stopping Cluster (pacemaker)... rh6-1: Stopping Cluster (pacemaker)... rh6-2: Stopping Cluster (pacemaker)... rh6-1: Successfully destroyed cluster rh6-2: Successfully destroyed cluster rh6-3: Successfully destroyed cluster Sending cluster config files to the nodes... rh6-1: Updated cluster.conf... rh6-2: Updated cluster.conf... rh6-3: Updated cluster.conf... Synchronizing pcsd certificates on nodes rh6-1, rh6-2, rh6-3... rh6-1: Success rh6-3: Success rh6-2: Success Restarting pcsd on the nodes in order to reload the certificates... rh6-1: Success rh6-3: Success rh6-2: Success rh6-2: Starting Cluster... rh6-3: Unable to connect to rh6-3 (Connection error) rh6-1: Unable to connect to rh6-1 (Connection error) Error: unable to start all nodes rh6-1: Unable to connect to rh6-1 (Connection error) rh6-3: Unable to connect to rh6-3 (Connection error)
Before Fix: [vm-rhel67-1 ~] $ rpm -q pcs pcs-0.9.155-1.el6.x86_64 Run it multiple times... [vm-rhel67-1 ~] $ pcs cluster setup --name=devcluster6 vm-rhel67-1 vm-rhel67-2 vm-rhel67-3 && pcs cluster start --all --wait Destroying cluster on nodes: vm-rhel67-1, vm-rhel67-2, vm-rhel67-3... vm-rhel67-1: Stopping Cluster (pacemaker)... vm-rhel67-3: Stopping Cluster (pacemaker)... vm-rhel67-2: Stopping Cluster (pacemaker)... vm-rhel67-1: Successfully destroyed cluster vm-rhel67-3: Successfully destroyed cluster vm-rhel67-2: Successfully destroyed cluster Sending cluster config files to the nodes... vm-rhel67-1: Updated cluster.conf... vm-rhel67-2: Updated cluster.conf... vm-rhel67-3: Updated cluster.conf... Synchronizing pcsd certificates on nodes vm-rhel67-1, vm-rhel67-2, vm-rhel67-3... vm-rhel67-1: Success vm-rhel67-2: Success vm-rhel67-3: Success Restarting pcsd on the nodes in order to reload the certificates... vm-rhel67-1: Success vm-rhel67-2: Success vm-rhel67-3: Success Exception in thread Thread-2: Traceback (most recent call last): File "/usr/lib64/python2.6/threading.py", line 532, in __bootstrap_inner self.run() File "/usr/lib64/python2.6/threading.py", line 484, in run self.__target(*self.__args, **self.__kwargs) File "/usr/lib/python2.6/site-packages/pcs/utils.py", line 1112, in worker returncode, output = action(node, *args, **kwargs) File "/usr/lib/python2.6/site-packages/pcs/utils.py", line 261, in startCluster return sendHTTPRequest(node, 'remote/cluster_start', None, False, not quiet) File "/usr/lib/python2.6/site-packages/pcs/utils.py", line 410, in sendHTTPRequest result = opener.open(url,data) File "/usr/lib64/python2.6/urllib2.py", line 391, in open response = self._open(req, data) File "/usr/lib64/python2.6/urllib2.py", line 409, in _open '_open', req) File "/usr/lib64/python2.6/urllib2.py", line 369, in _call_chain result = func(*args) File "/usr/lib64/python2.6/urllib2.py", line 1198, in https_open return self.do_open(httplib.HTTPSConnection, req) File "/usr/lib64/python2.6/urllib2.py", line 1163, in do_open r = h.getresponse() File "/usr/lib64/python2.6/httplib.py", line 1012, in getresponse response.begin() File "/usr/lib64/python2.6/httplib.py", line 404, in begin version, status, reason = self._read_status() File "/usr/lib64/python2.6/httplib.py", line 368, in _read_status raise BadStatusLine(line) BadStatusLine Exception in thread Thread-1: Traceback (most recent call last): File "/usr/lib64/python2.6/threading.py", line 532, in __bootstrap_inner self.run() File "/usr/lib64/python2.6/threading.py", line 484, in run self.__target(*self.__args, **self.__kwargs) File "/usr/lib/python2.6/site-packages/pcs/utils.py", line 1112, in worker returncode, output = action(node, *args, **kwargs) File "/usr/lib/python2.6/site-packages/pcs/utils.py", line 261, in startCluster return sendHTTPRequest(node, 'remote/cluster_start', None, False, not quiet) File "/usr/lib/python2.6/site-packages/pcs/utils.py", line 410, in sendHTTPRequest result = opener.open(url,data) File "/usr/lib64/python2.6/urllib2.py", line 391, in open response = self._open(req, data) File "/usr/lib64/python2.6/urllib2.py", line 409, in _open '_open', req) File "/usr/lib64/python2.6/urllib2.py", line 369, in _call_chain result = func(*args) File "/usr/lib64/python2.6/urllib2.py", line 1198, in https_open return self.do_open(httplib.HTTPSConnection, req) File "/usr/lib64/python2.6/urllib2.py", line 1163, in do_open r = h.getresponse() File "/usr/lib64/python2.6/httplib.py", line 1012, in getresponse response.begin() File "/usr/lib64/python2.6/httplib.py", line 404, in begin version, status, reason = self._read_status() File "/usr/lib64/python2.6/httplib.py", line 368, in _read_status raise BadStatusLine(line) BadStatusLine Exception in thread Thread-3: Traceback (most recent call last): File "/usr/lib64/python2.6/threading.py", line 532, in __bootstrap_inner self.run() File "/usr/lib64/python2.6/threading.py", line 484, in run self.__target(*self.__args, **self.__kwargs) File "/usr/lib/python2.6/site-packages/pcs/utils.py", line 1112, in worker returncode, output = action(node, *args, **kwargs) File "/usr/lib/python2.6/site-packages/pcs/utils.py", line 261, in startCluster return sendHTTPRequest(node, 'remote/cluster_start', None, False, not quiet) File "/usr/lib/python2.6/site-packages/pcs/utils.py", line 410, in sendHTTPRequest result = opener.open(url,data) File "/usr/lib64/python2.6/urllib2.py", line 391, in open response = self._open(req, data) File "/usr/lib64/python2.6/urllib2.py", line 409, in _open '_open', req) File "/usr/lib64/python2.6/urllib2.py", line 369, in _call_chain result = func(*args) File "/usr/lib64/python2.6/urllib2.py", line 1198, in https_open return self.do_open(httplib.HTTPSConnection, req) File "/usr/lib64/python2.6/urllib2.py", line 1163, in do_open r = h.getresponse() File "/usr/lib64/python2.6/httplib.py", line 1012, in getresponse response.begin() File "/usr/lib64/python2.6/httplib.py", line 404, in begin version, status, reason = self._read_status() File "/usr/lib64/python2.6/httplib.py", line 368, in _read_status raise BadStatusLine(line) BadStatusLine Waiting for node(s) to start... vm-rhel67-1: Error connecting to vm-rhel67-1 - (HTTP error: 400) vm-rhel67-3: Started vm-rhel67-2: Started Error: unable to verify all nodes have started After Fix: [vm-rhel67-1 ~] $ rpm -q pcs pcs-0.9.155-2.el6.x86_64 Run it multiple times... [vm-rhel67-1 ~] $ pcs cluster setup --name=devcluster6 vm-rhel67-1 vm-rhel67-2 vm-rhel67-3 && pcs cluster start --all --wait Destroying cluster on nodes: vm-rhel67-1, vm-rhel67-2, vm-rhel67-3... vm-rhel67-3: Stopping Cluster (pacemaker)... vm-rhel67-1: Stopping Cluster (pacemaker)... vm-rhel67-2: Stopping Cluster (pacemaker)... vm-rhel67-3: Successfully destroyed cluster vm-rhel67-1: Successfully destroyed cluster vm-rhel67-2: Successfully destroyed cluster Sending cluster config files to the nodes... vm-rhel67-1: Updated cluster.conf... vm-rhel67-2: Updated cluster.conf... vm-rhel67-3: Updated cluster.conf... Synchronizing pcsd certificates on nodes vm-rhel67-1, vm-rhel67-2, vm-rhel67-3... vm-rhel67-1: Success vm-rhel67-2: Success vm-rhel67-3: Success Restarting pcsd on the nodes in order to reload the certificates... vm-rhel67-1: Success vm-rhel67-2: Success vm-rhel67-3: Success vm-rhel67-2: Starting Cluster... vm-rhel67-1: Starting Cluster... vm-rhel67-3: Starting Cluster... Waiting for node(s) to start... vm-rhel67-2: Started vm-rhel67-3: Started vm-rhel67-1: Started
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHBA-2017-0707.html