We're seeing sporadic errors running update-cluster where the haproxy cartridge fails to read the primary cartridge's manifest because of a permission denied error: Failed to execute: 'control update-cluster' for /var/lib/openshift/5270b29b6cec0e97d200030b/haproxy /opt/rh/ruby193/root/usr/share/gems/gems/safe_yaml-0.9.1/lib/safe_yaml.rb:181:in `initialize': Permission denied - /var/lib/openshift/5270b29b6cec0e97d200030b/ruby/metadata/manifest.yml (Errno::EACCES) from /opt/rh/ruby193/root/usr/share/gems/gems/safe_yaml-0.9.1/lib/safe_yaml.rb:181:in `open' from /opt/rh/ruby193/root/usr/share/gems/gems/safe_yaml-0.9.1/lib/safe_yaml.rb:181:in `safe_load_file' from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-common-1.16.2/lib/openshift-origin-common/models/manifest.rb:217:in `initialize' from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.16.8/lib/openshift-origin-node/model/v2_cart_model.rb:168:in `new' from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.16.8/lib/openshift-origin-node/model/v2_cart_model.rb:168:in `get_cartridge_from_directory' from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.16.8/lib/openshift-origin-node/model/v2_cart_model.rb:94:in `primary_cartridge' from /usr/lib/openshift/cartridge_sdk/ruby/sdk.rb:26:in `primary_cartridge' from /usr/lib/openshift/cartridge_sdk/ruby/sdk.rb:30:in `primary_cartridge_manifest' from -e:1:in `<main>'
Commit pushed to master at https://github.com/openshift/origin-server https://github.com/openshift/origin-server/commit/65d5850ef1daec53f76adc522096df64d94237d9 Bug 1025043 Added logging to assist in debugging errors when haproxy's update-cluster control method fails.
https://github.com/openshift/origin-server/pull/4091
November 06 21:54:10 INFO 527b00749c35a15993000030 update-cluster against 'haproxy' November 06 21:54:16 INFO BZ1025043: Gear 527b00749c35a15993000030 - got exception running update-cluster for the proxy: Failed to execute: 'control update-cluster' for /var/lib/openshift/527b00749c35a15993000030/haproxy November 06 21:54:16 INFO BZ1025043: Gear 527b00749c35a15993000030 - directory listing of primary cartridge directory: November 06 21:54:16 INFO BZ1025043: Gear 527b00749c35a15993000030 - drwxr-xr-x. root 527b00749c35a15993000030 system_u:object_r:openshift_var_lib_t:s0:c0,c1001 . Change the manifest.yml permission and then try to scale up the app will get the above info in the platform.log. Move but to verified.
build version is devenv_3998