Bug 1305913 - pcs cluster setup may get stuck due to pipe issue
Summary: pcs cluster setup may get stuck due to pipe issue
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: pcs
Version: 6.8
Hardware: Unspecified
OS: Unspecified
medium
unspecified
Target Milestone: rc
: ---
Assignee: Tomas Jelinek
QA Contact: cluster-qe@redhat.com
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-02-09 14:59 UTC by Radek Steiger
Modified: 2016-05-10 19:27 UTC (History)
5 users (show)

Fixed In Version: pcs-0.9.148-4.el6
Doc Type: Bug Fix
Doc Text:
Cause: User runs 'pcs cluster setup' command. Consequence: Pcs gets stuck occasionally. Fix: Close file descriptors when running external processes from pcs. Result: Pcs does not get stuck.
Clone Of:
Environment:
Last Closed: 2016-05-10 19:27:22 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2016:0739 0 normal SHIPPED_LIVE pcs bug fix update 2016-05-10 22:29:32 UTC

Description Radek Steiger 2016-02-09 14:59:05 UTC
> Description of problem:

Running cluster setup may soemtimes get stuck with pcsd-cli.rb subprocesses never returning. It looks like the pipe between python caller and ruby subprocesses when calling for read_tokens.

This can be best reproduced by running the cluster setup in a loop with 10 seconds delay between the attempts (timing seems to be a key here). It might take several minutes before the setup will get stuck.


> Version-Release number of selected component (if applicable):

[root@virt-031 ~]# rpm -q pcs
pcs-0.9.148-3.el6.x86_64


> How reproducible:

~ 10%


> Steps to Reproduce:
1. while [ $? -eq 0 ]; do sleep 10; date; pcs cluster setup --name STSRHTS24987 virt-030 virt-031 virt-033 --force --debug; done
2. wait until the loop above hangs


> Actual results:

Debug output get stuck here:

<...snip...>
Destroying cluster on nodes: virt-030, virt-031, virt-033...
Running: /usr/bin/ruby -I/usr/lib/pcsd/ /usr/lib/pcsd/pcsd-cli.rb read_tokens
--Debug Input Start--
{}
--Debug Input End--
Running: /usr/bin/ruby -I/usr/lib/pcsd/ /usr/lib/pcsd/pcsd-cli.rb read_tokens
--Debug Input Start--
{}
--Debug Input End--
Running: /usr/bin/ruby -I/usr/lib/pcsd/ /usr/lib/pcsd/pcsd-cli.rb read_tokens
--Debug Input Start--
{}
--Debug Input End--

processes:

root	  6808  4.8  1.7 455108 18080 pts/3    Sl+  15:57   0:00          \_ /usr/bin/python /usr/sbin/pcs cluster setup --name STSRHTS24987 virt-030 virt-031 virt-033 --force --debug
root	  6983 11.0  0.6  41772  6756 pts/3    R+   15:57   0:00              \_ /usr/bin/ruby -I/usr/lib/pcsd/ /usr/lib/pcsd/pcsd-cli.rb read_tokens
root	  6985 11.0  0.6  41776  6756 pts/3    R+   15:57   0:00              \_ /usr/bin/ruby -I/usr/lib/pcsd/ /usr/lib/pcsd/pcsd-cli.rb read_tokens
root	  6986 11.0  0.6  41768  6760 pts/3    R+   15:57   0:00              \_ /usr/bin/ruby -I/usr/lib/pcsd/ /usr/lib/pcsd/pcsd-cli.rb read_tokens


> Expected results:

Cluster setup finishes in all cases.

Comment 5 Tomas Jelinek 2016-02-15 12:47:45 UTC
patch in upstream: https://github.com/feist/pcs/commit/59dde9ba191bc079aee08ac25576b51b9a85681c

test in bug description

Comment 6 Ivan Devat 2016-02-17 12:29:37 UTC
The problem occurs only occasionally, it is not easy to reproduce. Tested only successful cluster setup after fix.

Comment 10 errata-xmlrpc 2016-05-10 19:27:22 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-0739.html


Note You need to log in before you can comment on or make changes to this bug.