Bug 1020365 - [deploy] Providing invalid Git ref to `rhc deploy` corrupts app-deployments and breaks application
Summary: [deploy] Providing invalid Git ref to `rhc deploy` corrupts app-deployments a...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: OpenShift Online
Classification: Red Hat
Component: Containers
Version: 2.x
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: ---
Assignee: Dan Mace
QA Contact: libra bugs
URL:
Whiteboard:
: 1020862 (view as bug list)
Depends On: 1019646
Blocks:
TreeView+ depends on / blocked
 
Reported: 2013-10-17 14:12 UTC by Dan Mace
Modified: 2015-05-14 23:30 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2014-02-26 19:08:02 UTC
Target Upstream Version:


Attachments (Terms of Use)
Check git ref before running deploy (1.29 KB, application/mbox)
2013-10-17 17:30 UTC, Hiro Asari
no flags Details

Description Dan Mace 2013-10-17 14:12:40 UTC
Description of problem:

Passing an invalid Git ref to `rhc deploy` with an app configured for `--no-auto-deploy` corrupts the app-deployments directory and leaves the gear stopped and broken.

Version-Release number of selected component (if applicable):


How reproducible:

Create a nodejs 0.6 application with `--no-auto-deploy` and run `rhc deploy` with an invalid Git ref.


Steps to Reproduce:
1. rhc app-create nodeapp nodejs-0.6
2. rhc configure-app -a nodeapp --no-auto-deploy
3. push a change to the application to receive a new Git ref and verify that the app still works
4. rhc deploy bogusrefid -a nodeapp

Actual results:

A cryptic error message such as:

Deployment of git ref 'd015767a' in progress for application nodeapp ...
Stopping NodeJS cartridge
Saving away previously installed Node modules
An error occurred executing 'gear deploy' (exit code: 1)
Error message: Shell command '
set -xe;
shopt -s dotglob;
if [ "$(find objects -type f 2>/dev/null | wc -l)" -eq "0" ]; then
  exit 0;
fi
git archive --format=tar d015767a | (cd
/var/lib/openshift/525fe917a9b79abcce000030/app-deployments/2013-10-17_09-59-49.869/repo && tar --warning=no-timestamp -xf
-);
' returned an error. rc=1
stdout: 
stderr: + shopt -s dotglob
++ find objects -type f
++ wc -l
+ '[' 28 -eq 0 ']'
+ git archive --format=tar d015767a
+ cd /var/lib/openshift/525fe917a9b79abcce000030/app-deployments/2013-10-17_09-59-49.869/repo
+ tar --warning=no-timestamp -xf -
fatal: Not a valid object name
tar: This does not look like a tar archive
tar: Exiting with failure status due to previous errors

For more details about the problem, try running the command again with the '--trace' option.
Success


Additionally:
* The current deployment symlink is missing from ~/app-deployments/by-id which is an invalid state/gear corruption
* The gear is now in a stopped state


Expected results:

A meaningful error message and no deployment attempt.


Additional info:

Comment 1 Hiro Asari 2013-10-17 17:30:42 UTC
Created attachment 813462 [details]
Check git ref before running deploy

A straightforward approach is to check that the given git ref is a valid one before deploying. This patch does the job.

However, this is apparently insufficient. For some reason the error message is not getting on the extended SSH data channel. As a result, 'rhc' command still thinks that the deploy command succeeded:

$ bx bin/rhc deploy foobar -a rack
Deployment of git ref 'foobar' in progress for application rack ...
Git ref foobar is invalid
Success

Comment 2 Dan Mace 2013-10-17 18:08:04 UTC
https://github.com/openshift/origin-server/pull/3916

NOTE: Please test this issue using ssh commands directly against the gear, e.g. `ssh user@host "gear deploy bogusref"` since a current rhc bug will report success even when the gear command reports failure:

https://bugzilla.redhat.com/show_bug.cgi?id=1020473

Comment 3 Andy Goldstein 2013-10-18 12:35:33 UTC
*** Bug 1020862 has been marked as a duplicate of this bug. ***

Comment 4 Hiro Asari 2013-10-18 14:20:21 UTC
Adding Bug 1019646 as dependency, due to SSH handling of 'rhc'.

Comment 5 Meng Bo 2013-10-21 11:15:57 UTC
$ ssh 52650b4c2b21cea5e0000190.rhcloud.com "gear deploy aaaassss"
Git ref aaaassss is invalid


Checked on devenv_3921, issue has been fixed.


Note You need to log in before you can comment on or make changes to this bug.