If an app deployment ends up with an empty metadata.json file (possibly due to the gear hitting its quota just before the file is written), this can cause issues later with deployment operations such as pruning. The specific issue that a user saw this week was with a git push failing on prereceive: Pushing to ssh://UUID@HOST/~/git/APP.git/ Counting objects: 26, done. Delta compression using up to 8 threads. Compressing objects: 100% (18/18), done. Writing objects: 100% (18/18), 1.74 KiB, done. Total 18 (delta 13), reused 0 (delta 0) remote: Not stopping cartridge php because hot deploy is enabled remote: Repairing links for 1 deployments remote: undefined method `last' for nil:NilClass To ssh://UUID@HOST/~/git/APP.git/ ! [remote rejected] master -> master (pre-receive hook declined) error: failed to push some refs to 'ssh://UUID@HOST/~/git/APP.git/ As best I can tell, this error comes from this line in the prune_deployments method of node/lib/openshift-origin-node/model/application_container_ext/deployments.rb : activation_cutoff = deployment_metadata_for(deployment_datetime).activations.last Since metadata.json exists but is empty, "activations" doesn't exist in the metadata hash. It seems like we should be doing some validation when we load a metadata file, and treating it as invalid (or populating sane defaults) if certain criteria are not met.
If users run into this, I think that running this in the affected gear is a valid workaround: for x in $( find -L app-deployments -maxdepth 3 -name metadata.json -size 0 ); do rm -rf $( dirname $x ); done There could be cases where the bad metadata.json is not zero-length, though. For those cases, the user may need to inspect each file. I typical file will look something like this: {"git_ref":"master","git_sha1":null,"id":"a7347d5c","hot_deploy":null,"force_clean_build":null,"activations":[],"checksum":null} if "activations" is not present, an error like the one in the bug description can occur. If there are other problems, such as invalid json, it will also fail, but with a different error.
Commit pushed to master at https://github.com/openshift/origin-server https://github.com/openshift/origin-server/commit/a96ef04aa5a69db4e3d92c4bd6a6f4324bf50bcc Bug 1054403 - Reset empty metadata.json file * Use defaults if file is empty
Checked on devenv_4243, issue fixed. Reproduce step on devenv-stage_634: 1. Create app 2. Clear the content of the metadata.json file of the deployment 3. Modify the local repo and git push remote: Stopping PHP cartridge remote: [Mon Jan 20 02:08:59 2014] [warn] PassEnv variable SHELL was undefined remote: [Mon Jan 20 02:08:59 2014] [warn] PassEnv variable USER was undefined remote: [Mon Jan 20 02:08:59 2014] [warn] PassEnv variable LOGNAME was undefined remote: Waiting for stop to finish remote: 795: unexpected token at '' To ssh://52dccb29d2796c2ba2000007.rhcloud.com/~/git/app1.git/ ! [remote rejected] master -> master (pre-receive hook declined)