Bug 1054403

Summary: empty metadata.json breaks app deployment
Product: OpenShift Online Reporter: Andy Grimm <agrimm>
Component: ContainersAssignee: Jhon Honce <jhonce>
Status: CLOSED CURRENTRELEASE QA Contact: libra bugs <libra-bugs>
Severity: medium Docs Contact:
Priority: medium    
Version: 2.xCC: bmeng, jgoulding, xtian
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1061422 (view as bug list) Environment:
Last Closed: 2014-01-30 00:56:05 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1061422    

Description Andy Grimm 2014-01-16 18:31:29 UTC 
If an app deployment ends up with an empty metadata.json file (possibly due to the gear hitting its quota just before the file is written), this can cause issues later with deployment operations such as pruning.  The specific issue that a user saw this week was with a git push failing on prereceive:

Pushing to ssh://UUID@HOST/~/git/APP.git/
Counting objects: 26, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (18/18), done.
Writing objects: 100% (18/18), 1.74 KiB, done.
Total 18 (delta 13), reused 0 (delta 0)
remote: Not stopping cartridge php because hot deploy is enabled
remote: Repairing links for 1 deployments
remote: undefined method `last' for nil:NilClass
To ssh://UUID@HOST/~/git/APP.git/
 ! [remote rejected] master -> master (pre-receive hook declined)
error: failed to push some refs to 'ssh://UUID@HOST/~/git/APP.git/

As best I can tell, this error comes from this line in the prune_deployments method of node/lib/openshift-origin-node/model/application_container_ext/deployments.rb :

activation_cutoff = deployment_metadata_for(deployment_datetime).activations.last

Since metadata.json exists but is empty, "activations" doesn't exist in the metadata hash.

It seems like we should be doing some validation when we load a metadata file, and treating it as invalid (or populating sane defaults) if certain criteria are not met.
Comment 1 Andy Grimm 2014-01-16 18:50:49 UTC 
If users run into this, I think that running this in the affected gear is a valid workaround:

for x in $( find -L app-deployments -maxdepth 3 -name metadata.json -size 0 ); do rm -rf $( dirname $x ); done

There could be cases where the bad metadata.json is not zero-length, though.  For those cases, the user may need to inspect each file.  I typical file will look something like this:

{"git_ref":"master","git_sha1":null,"id":"a7347d5c","hot_deploy":null,"force_clean_build":null,"activations":[],"checksum":null}

if "activations" is not present, an error like the one in the bug description can occur.  If there are other problems, such as invalid json, it will also fail, but with a different error.
Comment 2 openshift-github-bot 2014-01-17 20:57:48 UTC 
Commit pushed to master at https://github.com/openshift/origin-server

https://github.com/openshift/origin-server/commit/a96ef04aa5a69db4e3d92c4bd6a6f4324bf50bcc
Bug 1054403 - Reset empty metadata.json file

* Use defaults if file is empty
Comment 3 Meng Bo 2014-01-20 08:09:35 UTC 
Checked on devenv_4243, issue fixed.

Reproduce step on devenv-stage_634:
 1. Create app
 2. Clear the content of the metadata.json file of the deployment
 3. Modify the local repo and git push

    remote: Stopping PHP cartridge
    remote: [Mon Jan 20 02:08:59 2014] [warn] PassEnv variable SHELL was undefined
    remote: [Mon Jan 20 02:08:59 2014] [warn] PassEnv variable USER was undefined
    remote: [Mon Jan 20 02:08:59 2014] [warn] PassEnv variable LOGNAME was undefined
    remote: Waiting for stop to finish
    remote: 795: unexpected token at ''
    To ssh://52dccb29d2796c2ba2000007.rhcloud.com/~/git/app1.git/
     ! [remote rejected] master -> master (pre-receive hook declined)