Bug 1025043

Summary: Sporadic permission denied errors trying to read primary cartridge's manifest.yml during update-cluster
Product: OpenShift Online Reporter: Andy Goldstein <agoldste>
Component: ImageAssignee: Andy Goldstein <agoldste>
Status: CLOSED CURRENTRELEASE QA Contact: libra bugs <libra-bugs>
Severity: medium Docs Contact:
Priority: medium    
Version: 2.xCC: bmeng, dmcphers, pmorie
Target Milestone: ---Keywords: UpcomingRelease
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-01-24 03:27:57 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Andy Goldstein 2013-10-30 19:57:27 UTC
We're seeing sporadic errors running update-cluster where the haproxy cartridge fails to read the primary cartridge's manifest because of a permission denied error:

Failed to execute: 'control update-cluster' for /var/lib/openshift/5270b29b6cec0e97d200030b/haproxy

/opt/rh/ruby193/root/usr/share/gems/gems/safe_yaml-0.9.1/lib/safe_yaml.rb:181:in `initialize': Permission denied - /var/lib/openshift/5270b29b6cec0e97d200030b/ruby/metadata/manifest.yml (Errno::EACCES)
        from /opt/rh/ruby193/root/usr/share/gems/gems/safe_yaml-0.9.1/lib/safe_yaml.rb:181:in `open'
        from /opt/rh/ruby193/root/usr/share/gems/gems/safe_yaml-0.9.1/lib/safe_yaml.rb:181:in `safe_load_file'
        from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-common-1.16.2/lib/openshift-origin-common/models/manifest.rb:217:in `initialize'
        from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.16.8/lib/openshift-origin-node/model/v2_cart_model.rb:168:in `new'
        from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.16.8/lib/openshift-origin-node/model/v2_cart_model.rb:168:in `get_cartridge_from_directory'
        from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.16.8/lib/openshift-origin-node/model/v2_cart_model.rb:94:in `primary_cartridge'
        from /usr/lib/openshift/cartridge_sdk/ruby/sdk.rb:26:in `primary_cartridge'
        from /usr/lib/openshift/cartridge_sdk/ruby/sdk.rb:30:in `primary_cartridge_manifest'
        from -e:1:in `<main>'

Comment 1 openshift-github-bot 2013-10-31 15:00:54 UTC
Commit pushed to master at https://github.com/openshift/origin-server

https://github.com/openshift/origin-server/commit/65d5850ef1daec53f76adc522096df64d94237d9
Bug 1025043

Added logging to assist in debugging errors when haproxy's
update-cluster control method fails.

Comment 2 Andy Goldstein 2013-11-05 17:37:17 UTC
https://github.com/openshift/origin-server/pull/4091

Comment 3 Meng Bo 2013-11-07 02:58:56 UTC
November 06 21:54:10 INFO 527b00749c35a15993000030 update-cluster against 'haproxy'
November 06 21:54:16 INFO BZ1025043: Gear 527b00749c35a15993000030 - got exception running update-cluster for the proxy: Failed to execute: 'control update-cluster' for /var/lib/openshift/527b00749c35a15993000030/haproxy
November 06 21:54:16 INFO BZ1025043: Gear 527b00749c35a15993000030 - directory listing of primary cartridge directory:
November 06 21:54:16 INFO BZ1025043: Gear 527b00749c35a15993000030 - drwxr-xr-x. root                     527b00749c35a15993000030 system_u:object_r:openshift_var_lib_t:s0:c0,c1001 .


Change the manifest.yml permission and then try to scale up the app will get the above info in the platform.log.

Move but to verified.

Comment 4 Meng Bo 2013-11-07 03:02:22 UTC
build version is devenv_3998