Bug 1267187 - engine-setup hangs indefinitely starting ovirt-websocket-proxy via service using python subprocess module
engine-setup hangs indefinitely starting ovirt-websocket-proxy via service us...
Status: CLOSED CURRENTRELEASE
Product: ovirt-engine
Classification: oVirt
Component: Services (Show other bugs)
3.5.5
Unspecified Unspecified
unspecified Severity high (vote)
: ovirt-3.5.6
: 3.5.6
Assigned To: Sandro Bonazzola
Karolína Hajná
integration
: Regression, ZStream
Depends On: 1266881
Blocks:
  Show dependency treegraph
 
Reported: 2015-09-29 05:23 EDT by Sandro Bonazzola
Modified: 2016-05-19 21:24 EDT (History)
8 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 1266881
Environment:
Last Closed: 2015-12-22 08:24:33 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: Integration
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
rule-engine: ovirt‑3.5.z+
rule-engine: blocker+
bmcclain: planning_ack+
sbonazzo: devel_ack+
pnovotny: testing_ack+


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
oVirt gerrit 46769 None None None Never
oVirt gerrit 46773 None None None Never
oVirt gerrit 46775 None None None Never
oVirt gerrit 46781 ovirt-engine-3.5 MERGED packaging: pythonlib: service: by default redirect to /dev/null stdout/stderr Never

  None (edit)
Description Sandro Bonazzola 2015-09-29 05:23:35 EDT
+++ This bug was initially created as a clone of Bug #1266881 +++

While installing ovirt-engine, the setup is stuck on "service ovirt-websocket-proxy start".
Looks like the service is not working as a proper daemon anymore.

Workaround: manually stop the service during the setup and re-start it when setup fnishes

Workaround: choose to not setup websocket proxy on the system while configuring ovirt-engine

--- Additional comment from Simone Tiraboschi on 2015-09-28 11:11:59 EDT ---

Adding details:

ovirt-websocket-proxy correctly starts if we manually invoke service

 [root@c66et1 ~]# /sbin/service ovirt-websocket-proxy start
 Starting oVirt Engine websockets proxy:                    [  OK  ]
 [root@c66et1 ~]# echo $?
 0

bu the issue happens if we start the service thought Otopi.
The python daemon goes on, service command exits but the python code doesn't notify it, service process is marked as defunct while the setup still monitors it:

29983  2913 29983  1416 pts/0    29983 Z+       0   0:00 [service] <defunct>

This few python lines are enough to reproduce it (manually stopping the service before that):
 import subprocess
 p = subprocess.Popen(('/sbin/service', 'ovirt-websocket-proxy', 'start'), stdin=None, stderr=subprocess.PIPE, stdout=subprocess.PIPE, close_fds=True,)
 output = p.communicate()
 print 'output: %s' % str(output)

service concludes, but this python script will wait forever on the communicate call.


If we run it with strace we see that:
 poll([{fd=3, events=POLLIN|POLLPRI}, {fd=5, events=POLLIN|POLLPRI}], 2, -1) = 1 ([{fd=3, revents=POLLIN}])
 read(3, "Starting oVirt Engine websockets"..., 4096) = 40
 poll([{fd=3, events=POLLIN|POLLPRI}, {fd=5, events=POLLIN|POLLPRI}], 2, -1) = 1 ([{fd=3, revents=POLLIN}])
 read(3, "\33[60G[\33[0;32m  OK  \33[0;39m]\r\n", 4096) = 29
 poll([{fd=3, events=POLLIN|POLLPRI}, {fd=5, events=POLLIN|POLLPRI}], 2, -1) = ? ERESTART_RESTARTBLOCK (Interrupted by signal)
 --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=10928, si_status=0, si_utime=0, si_stime=0} ---
 restart_syscall(<... resuming interrupted call ...>

The python script correctly gets its SIGCHLD when the service process exits but no code got executed and it continue to wait. And it will wait indefinitely cause at that point service is already died.

It could be related to this one:
https://bugzilla.redhat.com/1065537

--- Additional comment from Simone Tiraboschi on 2015-09-28 11:16:27 EDT ---

Seen with python 2.6.6-64.el6

--- Additional comment from Alon Bar-Lev on 2015-09-28 15:05:27 EDT ---

not sure I understand, if it is a bug in python and a regression, what version of python last work, what is the first that does not?

--- Additional comment from Alon Bar-Lev on 2015-09-28 15:21:52 EDT ---

Checkout the service check I submitted, it closes stdout/stderr of caller, should resolve this issue, I have no el6 environment to test.

--- Additional comment from Simone Tiraboschi on 2015-09-28 16:18:33 EDT ---

(In reply to Alon Bar-Lev from comment #4)
> Checkout the service check I submitted, it closes stdout/stderr of caller,
> should resolve this issue, I have no el6 environment to test.

It works on el6, thanks.

--- Additional comment from Michal Skrivanek on 2015-09-29 03:45:12 EDT ---

since this is a regression caused by el6 python we need to backport it to 3.5.z as well
Comment 1 Red Hat Bugzilla Rules Engine 2015-09-29 05:33:39 EDT
This bug report has Keywords: Regression or TestBlocker.
Since no regressions or test blockers are allowed between releases, it is also being identified as a blocker for this release. Please resolve ASAP.
Comment 2 Karolína Hajná 2015-11-04 06:24:47 EST
Verified on vt18 (rhevm-3.5.6.1-0.1.el6ev.noarch)
Comment 3 Sandro Bonazzola 2015-12-22 08:24:33 EST
oVirt 3.5.6 has been released and the bz verified, moving to closed current release.

Note You need to log in before you can comment on or make changes to this bug.