+++ This bug was initially created as a clone of Bug #1054403 +++ If an app deployment ends up with an empty metadata.json file (possibly due to the gear hitting its quota just before the file is written), this can cause issues later with deployment operations such as pruning. The specific issue that a user saw this week was with a git push failing on prereceive: Pushing to ssh://UUID@HOST/~/git/APP.git/ Counting objects: 26, done. Delta compression using up to 8 threads. Compressing objects: 100% (18/18), done. Writing objects: 100% (18/18), 1.74 KiB, done. Total 18 (delta 13), reused 0 (delta 0) remote: Not stopping cartridge php because hot deploy is enabled remote: Repairing links for 1 deployments remote: undefined method `last' for nil:NilClass To ssh://UUID@HOST/~/git/APP.git/ ! [remote rejected] master -> master (pre-receive hook declined) error: failed to push some refs to 'ssh://UUID@HOST/~/git/APP.git/ As best I can tell, this error comes from this line in the prune_deployments method of node/lib/openshift-origin-node/model/application_container_ext/deployments.rb : activation_cutoff = deployment_metadata_for(deployment_datetime).activations.last Since metadata.json exists but is empty, "activations" doesn't exist in the metadata hash. It seems like we should be doing some validation when we load a metadata file, and treating it as invalid (or populating sane defaults) if certain criteria are not met. --- Additional comment from Andy Grimm on 2014-01-16 13:50:49 EST --- If users run into this, I think that running this in the affected gear is a valid workaround: for x in $( find -L app-deployments -maxdepth 3 -name metadata.json -size 0 ); do rm -rf $( dirname $x ); done There could be cases where the bad metadata.json is not zero-length, though. For those cases, the user may need to inspect each file. I typical file will look something like this: {"git_ref":"master","git_sha1":null,"id":"a7347d5c","hot_deploy":null,"force_clean_build":null,"activations":[],"checksum":null} if "activations" is not present, an error like the one in the bug description can occur. If there are other problems, such as invalid json, it will also fail, but with a different error. --- Additional comment from openshift-github-bot on 2014-01-17 15:57:48 EST --- Commit pushed to master at https://github.com/openshift/origin-server https://github.com/openshift/origin-server/commit/a96ef04aa5a69db4e3d92c4bd6a6f4324bf50bcc Bug 1054403 - Reset empty metadata.json file * Use defaults if file is empty --- Additional comment from Meng Bo on 2014-01-20 03:09:35 EST --- Checked on devenv_4243, issue fixed. Reproduce step on devenv-stage_634: 1. Create app 2. Clear the content of the metadata.json file of the deployment 3. Modify the local repo and git push remote: Stopping PHP cartridge remote: [Mon Jan 20 02:08:59 2014] [warn] PassEnv variable SHELL was undefined remote: [Mon Jan 20 02:08:59 2014] [warn] PassEnv variable USER was undefined remote: [Mon Jan 20 02:08:59 2014] [warn] PassEnv variable LOGNAME was undefined remote: Waiting for stop to finish remote: 795: unexpected token at '' To ssh://52dccb29d2796c2ba2000007.rhcloud.com/~/git/app1.git/ ! [remote rejected] master -> master (pre-receive hook declined)
The bug is verified on puddle-02-06 with following actions: 1.clear the metadata.json. [php53-hanli1dom.oseanli.cn 2014-02-07_00-16-55.753]\> >metadata.json 2.Modify file under local repo of php53. 3.git push [ose215@dhcp-9-237 php53]$ gitpush [master 65ae73e] a 1 files changed, 1 insertions(+), 1 deletions(-) Counting objects: 5, done. Delta compression using up to 8 threads. Compressing objects: 100% (2/2), done. Writing objects: 100% (3/3), 255 bytes, done. Total 3 (delta 1), reused 0 (delta 0) remote: Stopping PHP cartridge remote: httpd: Could not reliably determine the server's fully qualified domain name, using nd216.oseanli.cn for ServerName remote: Waiting for stop to finish remote: Building git ref 'master', commit 65ae73e remote: Building PHP cartridge remote: Preparing build for deployment remote: Deployment id is 203fcde4 remote: Activating deployment remote: Starting PHP cartridge remote: httpd: Could not reliably determine the server's fully qualified domain name, using nd216.oseanli.cn for ServerName remote: ------------------------- remote: Git Post-Receive Result: success remote: Activation status: success remote: Deployment completed with status: success To ssh://52e9f9c84945dc0f61000181.cn/~/git/php53.git/ fa7ff3c..65ae73e master -> master
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHBA-2014-0209.html