Bug 1061422

+++ This bug was initially created as a clone of Bug #1054403 +++

If an app deployment ends up with an empty metadata.json file (possibly due to the gear hitting its quota just before the file is written), this can cause issues later with deployment operations such as pruning.  The specific issue that a user saw this week was with a git push failing on prereceive:

Pushing to ssh://UUID@HOST/~/git/APP.git/
Counting objects: 26, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (18/18), done.
Writing objects: 100% (18/18), 1.74 KiB, done.
Total 18 (delta 13), reused 0 (delta 0)
remote: Not stopping cartridge php because hot deploy is enabled
remote: Repairing links for 1 deployments
remote: undefined method `last' for nil:NilClass
To ssh://UUID@HOST/~/git/APP.git/
 ! [remote rejected] master -> master (pre-receive hook declined)
error: failed to push some refs to 'ssh://UUID@HOST/~/git/APP.git/

As best I can tell, this error comes from this line in the prune_deployments method of node/lib/openshift-origin-node/model/application_container_ext/deployments.rb :

activation_cutoff = deployment_metadata_for(deployment_datetime).activations.last

Since metadata.json exists but is empty, "activations" doesn't exist in the metadata hash.

It seems like we should be doing some validation when we load a metadata file, and treating it as invalid (or populating sane defaults) if certain criteria are not met.

--- Additional comment from Andy Grimm on 2014-01-16 13:50:49 EST ---

If users run into this, I think that running this in the affected gear is a valid workaround:

for x in $( find -L app-deployments -maxdepth 3 -name metadata.json -size 0 ); do rm -rf $( dirname $x ); done

There could be cases where the bad metadata.json is not zero-length, though.  For those cases, the user may need to inspect each file.  I typical file will look something like this:

{"git_ref":"master","git_sha1":null,"id":"a7347d5c","hot_deploy":null,"force_clean_build":null,"activations":[],"checksum":null}

if "activations" is not present, an error like the one in the bug description can occur.  If there are other problems, such as invalid json, it will also fail, but with a different error.

--- Additional comment from openshift-github-bot on 2014-01-17 15:57:48 EST ---

Commit pushed to master at https://github.com/openshift/origin-server

https://github.com/openshift/origin-server/commit/a96ef04aa5a69db4e3d92c4bd6a6f4324bf50bcc
Bug 1054403 - Reset empty metadata.json file

* Use defaults if file is empty

--- Additional comment from Meng Bo on 2014-01-20 03:09:35 EST ---

Checked on devenv_4243, issue fixed.

Reproduce step on devenv-stage_634:
 1. Create app
 2. Clear the content of the metadata.json file of the deployment
 3. Modify the local repo and git push

    remote: Stopping PHP cartridge
    remote: [Mon Jan 20 02:08:59 2014] [warn] PassEnv variable SHELL was undefined
    remote: [Mon Jan 20 02:08:59 2014] [warn] PassEnv variable USER was undefined
    remote: [Mon Jan 20 02:08:59 2014] [warn] PassEnv variable LOGNAME was undefined
    remote: Waiting for stop to finish
    remote: 795: unexpected token at ''
    To ssh://52dccb29d2796c2ba2000007.rhcloud.com/~/git/app1.git/
     ! [remote rejected] master -> master (pre-receive hook declined)