Bug 1020365

Summary: [deploy] Providing invalid Git ref to `rhc deploy` corrupts app-deployments and breaks application
Product: OpenShift Online Reporter: Dan Mace <dmace>
Component: ContainersAssignee: Dan Mace <dmace>
Status: CLOSED CURRENTRELEASE QA Contact: libra bugs <libra-bugs>
Severity: medium Docs Contact:
Priority: medium    
Version: 2.xCC: bmeng, jkeck, xtian
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-02-26 19:08:02 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1019646    
Bug Blocks:    
Attachments:
Description Flags
Check git ref before running deploy none

Description Dan Mace 2013-10-17 14:12:40 UTC
Description of problem:

Passing an invalid Git ref to `rhc deploy` with an app configured for `--no-auto-deploy` corrupts the app-deployments directory and leaves the gear stopped and broken.

Version-Release number of selected component (if applicable):


How reproducible:

Create a nodejs 0.6 application with `--no-auto-deploy` and run `rhc deploy` with an invalid Git ref.


Steps to Reproduce:
1. rhc app-create nodeapp nodejs-0.6
2. rhc configure-app -a nodeapp --no-auto-deploy
3. push a change to the application to receive a new Git ref and verify that the app still works
4. rhc deploy bogusrefid -a nodeapp

Actual results:

A cryptic error message such as:

Deployment of git ref 'd015767a' in progress for application nodeapp ...
Stopping NodeJS cartridge
Saving away previously installed Node modules
An error occurred executing 'gear deploy' (exit code: 1)
Error message: Shell command '
set -xe;
shopt -s dotglob;
if [ "$(find objects -type f 2>/dev/null | wc -l)" -eq "0" ]; then
  exit 0;
fi
git archive --format=tar d015767a | (cd
/var/lib/openshift/525fe917a9b79abcce000030/app-deployments/2013-10-17_09-59-49.869/repo && tar --warning=no-timestamp -xf
-);
' returned an error. rc=1
stdout: 
stderr: + shopt -s dotglob
++ find objects -type f
++ wc -l
+ '[' 28 -eq 0 ']'
+ git archive --format=tar d015767a
+ cd /var/lib/openshift/525fe917a9b79abcce000030/app-deployments/2013-10-17_09-59-49.869/repo
+ tar --warning=no-timestamp -xf -
fatal: Not a valid object name
tar: This does not look like a tar archive
tar: Exiting with failure status due to previous errors

For more details about the problem, try running the command again with the '--trace' option.
Success


Additionally:
* The current deployment symlink is missing from ~/app-deployments/by-id which is an invalid state/gear corruption
* The gear is now in a stopped state


Expected results:

A meaningful error message and no deployment attempt.


Additional info:

Comment 1 Hiro Asari 2013-10-17 17:30:42 UTC
Created attachment 813462 [details]
Check git ref before running deploy

A straightforward approach is to check that the given git ref is a valid one before deploying. This patch does the job.

However, this is apparently insufficient. For some reason the error message is not getting on the extended SSH data channel. As a result, 'rhc' command still thinks that the deploy command succeeded:

$ bx bin/rhc deploy foobar -a rack
Deployment of git ref 'foobar' in progress for application rack ...
Git ref foobar is invalid
Success

Comment 2 Dan Mace 2013-10-17 18:08:04 UTC
https://github.com/openshift/origin-server/pull/3916

NOTE: Please test this issue using ssh commands directly against the gear, e.g. `ssh user@host "gear deploy bogusref"` since a current rhc bug will report success even when the gear command reports failure:

https://bugzilla.redhat.com/show_bug.cgi?id=1020473

Comment 3 Andy Goldstein 2013-10-18 12:35:33 UTC
*** Bug 1020862 has been marked as a duplicate of this bug. ***

Comment 4 Hiro Asari 2013-10-18 14:20:21 UTC
Adding Bug 1019646 as dependency, due to SSH handling of 'rhc'.

Comment 5 Meng Bo 2013-10-21 11:15:57 UTC
$ ssh 52650b4c2b21cea5e0000190.rhcloud.com "gear deploy aaaassss"
Git ref aaaassss is invalid


Checked on devenv_3921, issue has been fixed.