Bug 987392

Summary: "Permission denied" debug info is found in scale_events.log from scaled up gears
Product: OpenShift Online Reporter: Jianwei Hou <jhou>
Component: ContainersAssignee: Mrunal Patel <mpatel>
Status: CLOSED CURRENTRELEASE QA Contact: libra bugs <libra-bugs>
Severity: medium Docs Contact:
Priority: medium    
Version: 2.xCC: rchopra
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-08-07 22:56:03 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Jianwei Hou 2013-07-23 10:42:45 UTC 
Description of problem:
Given a scalable app has scaled to 2 gears, when it scales up to the 3rd gear, the "stdout: /var/lib/openshift/51ee31813b2c860a4100000f/haproxy//usr/bin/remove-gear:34:in `read': Permission denied" will be found from the scale_events.log in the 2nd gear.

Version-Release number of selected component (if applicable):
On devenv_3540

How reproducible:
Always

Steps to Reproduce:
1. Create scalable php application, scale it up to 2 gears
2. Scale up again to the 3rd gear, in the meantime, tailf scale_events.log from both gears in rhcsh


Actual results:
The following error can be found from the scale_events.log from the 2nd gear
<--------------------------------------->
D, [2013-07-23T03:36:59.181589 #11172] DEBUG -- : GEAR_DOWN - remove-gear: exit: pid 13296 exit 1  stdout: /var/lib/openshift/51ee31813b2c860a4100000f/haproxy//usr/bin/remove-gear:34:in `read': Permission denied - /var/lib/openshift/8f96d02cf36911e2aeb722000a975b3b/.auth/token (Errno::EACCES)
	from /var/lib/openshift/51ee31813b2c860a4100000f/haproxy//usr/bin/remove-gear:34:in `initialize'
	from /var/lib/openshift/51ee31813b2c860a4100000f/haproxy//usr/bin/remove-gear:207:in `new'
	from /var/lib/openshift/51ee31813b2c860a4100000f/haproxy//usr/bin/remove-gear:207:in `<main>'
<--------------------------------------->

In above log, "8f96d02cf36911e2aeb722000a975b3b" is the head gear, "51ee31813b2c860a4100000f" is the 2nd gear created on scale up event.

Expected results:
Should not see above errors

Additional info:
Giving a low severity since it does not affect any funtionality
Comment 1 Abhishek Gupta 2013-07-23 19:39:36 UTC 
Is this a case of HAProxy trying to scale down in the absence of traffic while the user is trying to scale up manually?
Comment 2 Jianwei Hou 2013-07-24 07:03:04 UTC 
(In reply to Abhishek Gupta from comment #1)
> Is this a case of HAProxy trying to scale down in the absence of traffic
> while the user is trying to scale up manually?

Yeah, should be this case.
Comment 3 Rajat Chopra 2013-07-29 19:17:52 UTC 
Fix in pull request https://github.com/openshift/origin-server/pull/3205
Comment 4 Jianwei Hou 2013-07-30 03:20:33 UTC 
The errors are not seen again after above PR is merged. Mark as verified.