Bug 1009017

Summary:	Gears are started that have stoplock set
Product:	OpenShift Online	Reporter:	Sten Turpin <sten>
Component:	Containers	Assignee:	Dan Mace <dmace>
Status:	CLOSED UPSTREAM	QA Contact:	libra bugs <libra-bugs>
Severity:	medium	Docs Contact:
Priority:	medium
Version:	2.x	CC:	jhonce, twiest
Target Milestone:	---	Keywords:	UpcomingRelease
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2013-09-17 18:25:32 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Sten Turpin 2013-09-17 13:50:03 UTC

Description of problem: gears with stoplock set are sometimes found running on nodes

Version-Release number of selected component (if applicable):
unsure

How reproducible:
sometimes

Steps to Reproduce:
1. Set a stoplock on a gear - unsure if this is happening via gear migration, user interaction, something else

Actual results:
Gear is found running: 

Expected results:
Gears with stoplock set should not be found running. 

Additional info:
USER       PID  PPID STAT %CPU WCHAN                                ELAPSED COMMAND
  6739      8525 24466 D     0.0 sync_page                           01:43:57 httpd
  6739      9750 24466 D     0.0 sync_page                           01:43:34 httpd
  6739     10845 24466 D     0.0 sync_page                           01:43:26 httpd
  6739     11066 24466 D     0.0 sync_page                           01:43:18 httpd
  6739     11724 24466 D     0.0 sync_page                           01:42:54 httpd
  6739     11874 24466 D     0.0 sync_page                           01:42:49 httpd
  6739     11885 24466 D     0.0 sync_page                           01:42:48 httpd
  6739     11886 24466 D     0.0 sync_page                           01:42:48 httpd
  6739     12929 24466 D     0.0 sync_page                           01:42:33 httpd
  6739     13347 24466 D     0.0 sync_page                           01:36:35 httpd
  6739     13492 24466 D     0.0 sync_page                           01:42:30 httpd
  6739     15417 24466 D     0.0 sync_page                           01:36:23 httpd
  6739     15865 24466 D     0.0 sync_page                           01:36:19 httpd
  6739     15938 24466 D     0.0 sync_page                           01:36:18 httpd
  6739     16119 24466 D     0.0 sync_page                           01:36:18 httpd
  6739     16343 24466 D     0.0 sync_page                           01:36:14 httpd
  6739     17142 24466 D     0.0 sync_page                           01:41:56 httpd
  6739     17510 24466 D     0.0 sync_page                           01:41:54 httpd
  6739     19121 24466 D     0.0 sync_page                           01:35:42 httpd
  6739     19261 24466 D     0.0 sync_page                           01:41:38 httpd
  6739     22043 24466 D     0.0 sync_page                           01:41:21 httpd
  6739     22198 24466 D     0.0 sync_page                           01:41:20 httpd
  6739     22996 24466 D     0.0 sync_page                           01:41:08 httpd
  6739     22997 24466 D     0.0 sync_page                           01:41:08 httpd
  6739     23356 24466 D     0.0 sync_page                           01:40:55 httpd
  6739     23370 24466 D     0.0 sync_page                           01:40:54 httpd
  6739     23371 24466 D     0.0 sync_page                           01:40:54 httpd
  6739     23372 24466 D     1.1 mem_cgroup_handle_oom               01:40:53 httpd
  6739     23375 24466 D     0.0 sync_page                           01:40:52 httpd
  6739     24999 24466 D     0.1 sync_page                           01:46:35 httpd
  6739     28342 24466 D     0.0 sync_page                           01:33:39 httpd
  6739     32362 24466 D     0.0 sync_page                           01:45:01 httpd

$ grep 6739 /etc/passwd
52382815e0b8cda7c800010f:x:6739:6739:OpenShift guest:/var/lib/openshift/52382815e0b8cda7c800010f:/usr/bin/oo-trap-user
$ sudo oo-admin-ctl-gears stopgear 52382815e0b8cda7c800010f
Gear is locked: 52382815e0b8cda7c800010f

Comment 1 Jhon Honce 2013-09-17 18:25:32 UTC

Note the line below:

USER       PID  PPID STAT %CPU WCHAN                                ELAPSED COMMAND
  6739     23372 24466 D     1.1 mem_cgroup_handle_oom               01:40:53 httpd

The httpd process failed to be stopped because it's stuck in disk wait trying to handle a OOM event.


Please track progress here https://trello.com/c/5A50jd0o