Bug 964251

Summary: Custom cartridge with syntax error in control script (!#/bin/bash) doesn't fail
Product: OKD Reporter: Clayton Coleman <ccoleman>
Component: PodAssignee: Abhishek Gupta <abhgupta>
Status: CLOSED CURRENTRELEASE QA Contact: libra bugs <libra-bugs>
Severity: high Docs Contact:
Priority: high    
Version: 2.xCC: jhou, xtian
Target Milestone: ---Keywords: Reopened
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: devenv_3257 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-06-11 03:59:29 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Example cartridge demonstrating the problem none

Description Clayton Coleman 2013-05-17 16:21:00 UTC
Created attachment 749403 [details]
Example cartridge demonstrating the problem

Gear is not correctly started, but not rolled back.

The attached file has a syntax error in the control script at line 112.  I can create the app, but then the cart isn't rolled back.  Node tries to start (fails) then tries to stop (fails), then the gear is left created

From platform-trace.log

D, [2013-05-17T12:17:45.451969 #1106] DEBUG -- : oo_spawn running /sbin/runuser -m -s /bin/sh 51965817e6608f0d63000004 -c "exec /usr/bin/runcon 'unconfined_u:system_r:openshift_t:s0:c0,c501' /bin/sh -c \"set -xe;
shopt -s dotglob;
rm -rf /var/lib/openshift/51965817e6608f0d63000004/app-root/runtime/repo/*;
git archive --format=tar HEAD | (cd /var/lib/openshift/51965817e6608f0d63000004/app-root/runtime/repo && tar --warning=no-timestamp -xf -);
\"": {:unsetenv_others=>false, :close_others=>true, :in=>"/dev/null", :chdir=>"/var/lib/openshift/51965817e6608f0d63000004/git/go.git", :out=>#<IO:fd 11>, :err=>#<IO:fd 9>}
D, [2013-05-17T12:17:46.535465 #1106] DEBUG -- : oo_spawn buffer(7/) + shopt -s dotglob

D, [2013-05-17T12:17:46.535857 #1106] DEBUG -- : oo_spawn buffer(7/) + rm -rf '/var/lib/openshift/51965817e6608f0d63000004/app-root/runtime/repo/*'

D, [2013-05-17T12:17:46.539170 #1106] DEBUG -- : oo_spawn buffer(7/) + git archive --format=tar HEAD

D, [2013-05-17T12:17:46.539738 #1106] DEBUG -- : oo_spawn buffer(7/) + cd /var/lib/openshift/51965817e6608f0d63000004/app-root/runtime/repo

D, [2013-05-17T12:17:46.540019 #1106] DEBUG -- : oo_spawn buffer(7/) + tar --warning=no-timestamp -xf -

D, [2013-05-17T12:17:46.610999 #1106] DEBUG -- : oo_spawn running service openshift-node-web-proxy reload: {:unsetenv_others=>false, :close_others=>true, :in=>"/dev/null", :out=>#<IO:fd 14>, :err=>#<IO:fd 11>}
D, [2013-05-17T12:17:46.672275 #1106] DEBUG -- : oo_spawn buffer(13/) Reloading node-web-proxy: 
D, [2013-05-17T12:17:46.675349 #1106] DEBUG -- : oo_spawn buffer(13/) [
D, [2013-05-17T12:17:46.675611 #1106] DEBUG -- : oo_spawn buffer(13/)   OK  
D, [2013-05-17T12:17:46.675877 #1106] DEBUG -- : oo_spawn buffer(13/) ]
D, [2013-05-17T12:17:46.676076 #1106] DEBUG -- : oo_spawn buffer(13/) 
D, [2013-05-17T12:17:46.676283 #1106] DEBUG -- : oo_spawn buffer(13/) 

D, [2013-05-17T12:17:46.680136 #1106] DEBUG -- : oo_spawn running /usr/sbin/httxt2dbm -f DB -i /etc/httpd/conf.d/openshift/nodes.txt -o /etc/httpd/conf.d/openshift/nodes.db-20130517-1106-1nxf5mi/new.db: {:unsetenv_others=>false, :close_others=>true, :in=>"/dev/null", :out=>#<IO:fd 13>, :err=>#<IO:fd 10>}
D, [2013-05-17T12:17:46.739804 #1106] DEBUG -- : oo_spawn running /usr/sbin/httxt2dbm -f DB -i /etc/httpd/conf.d/openshift/nodes.txt -o /etc/httpd/conf.d/openshift/nodes.db-20130517-1106-34uzra/new.db: {:unsetenv_others=>false, :close_others=>true, :in=>"/dev/null", :out=>#<IO:fd 13>, :err=>#<IO:fd 10>}
D, [2013-05-17T12:17:46.789648 #1106] DEBUG -- : oo_spawn running cgset -r cpu.cfs_quota_us=30000 /openshift/51965817e6608f0d63000004: {:unsetenv_others=>false, :close_others=>true, :in=>"/dev/null", :out=>#<IO:fd 11>, :err=>#<IO:fd 9>}
D, [2013-05-17T12:17:48.100255 #1106] DEBUG -- : oo_spawn running cgget -a /openshift/51965817e6608f0d63000004 >/dev/null: {:unsetenv_others=>false, :close_others=>true, :in=>"/dev/null", :out=>#<IO:fd 11>, :err=>#<IO:fd 9>}
D, [2013-05-17T12:17:48.150540 #1106] DEBUG -- : oo_spawn running cgget -n -v -r cpu.cfs_period_us /openshift/51965817e6608f0d63000004: {:unsetenv_others=>false, :close_others=>true, :in=>"/dev/null", :out=>#<IO:fd 11>, :err=>#<IO:fd 9>}
D, [2013-05-17T12:17:48.184420 #1106] DEBUG -- : oo_spawn buffer(10/) 100000

D, [2013-05-17T12:17:48.185978 #1106] DEBUG -- : oo_spawn running cgget -n -v -r cpu.cfs_quota_us /openshift/51965817e6608f0d63000004: {:unsetenv_others=>false, :close_others=>true, :in=>"/dev/null", :out=>#<IO:fd 11>, :err=>#<IO:fd 9>}
D, [2013-05-17T12:17:48.219885 #1106] DEBUG -- : oo_spawn buffer(10/) 30000

D, [2013-05-17T12:17:48.221164 #1106] DEBUG -- : oo_spawn running cgset -r cpu.cfs_quota_us=100000 /openshift/51965817e6608f0d63000004: {:unsetenv_others=>false, :close_others=>true, :in=>"/dev/null", :out=>#<IO:fd 11>, :err=>#<IO:fd 9>}
D, [2013-05-17T12:17:48.470954 #1106] DEBUG -- : oo_spawn running /sbin/runuser -m -s /bin/sh 51965817e6608f0d63000004 -c "exec /usr/bin/runcon 'unconfined_u:system_r:openshift_t:s0:c0,c501' /bin/sh -c \"set -e; /var/lib/openshift/51965817e6608f0d63000004/go/bin/control start\"": {:unsetenv_others=>true, :close_others=>true, :in=>"/dev/null", :chdir=>"/var/lib/openshift/51965817e6608f0d63000004/", :out=>#<IO:fd 11>, :err=>#<IO:fd 9>}
D, [2013-05-17T12:17:49.602711 #1106] DEBUG -- : oo_spawn buffer(7/) /var/lib/openshift/51965817e6608f0d63000004/go/bin/control: line 113: syntax error near unexpected token `}'

D, [2013-05-17T12:17:49.603023 #1106] DEBUG -- : oo_spawn buffer(7/) /var/lib/openshift/51965817e6608f0d63000004/go/bin/control: line 113: `}'

D, [2013-05-17T12:17:49.608961 #1106] DEBUG -- : oo_spawn running cgset -r cpu.cfs_quota_us=30000 /openshift/51965817e6608f0d63000004: {:unsetenv_others=>false, :close_others=>true, :in=>"/dev/null", :out=>#<IO:fd 11>, :err=>#<IO:fd 9>}
D, [2013-05-17T12:17:49.863309 #1106] DEBUG -- : oo_spawn running cgget -a /openshift/51965817e6608f0d63000004 >/dev/null: {:unsetenv_others=>false, :close_others=>true, :in=>"/dev/null", :out=>#<IO:fd 11>, :err=>#<IO:fd 9>}
D, [2013-05-17T12:17:49.928025 #1106] DEBUG -- : oo_spawn running cgget -n -v -r cpu.cfs_period_us /openshift/51965817e6608f0d63000004: {:unsetenv_others=>false, :close_others=>true, :in=>"/dev/null", :out=>#<IO:fd 11>, :err=>#<IO:fd 9>}
D, [2013-05-17T12:17:49.962161 #1106] DEBUG -- : oo_spawn buffer(10/) 100000

D, [2013-05-17T12:17:49.967880 #1106] DEBUG -- : oo_spawn running cgget -n -v -r cpu.cfs_quota_us /openshift/51965817e6608f0d63000004: {:unsetenv_others=>false, :close_others=>true, :in=>"/dev/null", :out=>#<IO:fd 11>, :err=>#<IO:fd 9>}
D, [2013-05-17T12:17:50.006176 #1106] DEBUG -- : oo_spawn buffer(10/) 30000

D, [2013-05-17T12:17:50.007569 #1106] DEBUG -- : oo_spawn running cgset -r cpu.cfs_quota_us=100000 /openshift/51965817e6608f0d63000004: {:unsetenv_others=>false, :close_others=>true, :in=>"/dev/null", :out=>#<IO:fd 11>, :err=>#<IO:fd 9>}
D, [2013-05-17T12:17:50.266417 #1106] DEBUG -- : oo_spawn running /sbin/runuser -m -s /bin/sh 51965817e6608f0d63000004 -c "exec /usr/bin/runcon 'unconfined_u:system_r:openshift_t:s0:c0,c501' /bin/sh -c \"set -e; /var/lib/openshift/51965817e6608f0d63000004/go/bin/control stop\"": {:unsetenv_others=>true, :close_others=>true, :in=>"/dev/null", :chdir=>"/var/lib/openshift/51965817e6608f0d63000004/", :out=>#<IO:fd 11>, :err=>#<IO:fd 9>}
D, [2013-05-17T12:17:51.423032 #1106] DEBUG -- : oo_spawn buffer(7/) /var/lib/openshift/51965817e6608f0d63000004/go/bin/control: line 113: syntax error near unexpected token `}'
/var/lib/openshift/51965817e6608f0d63000004/go/bin/control: line 113: `}'

D, [2013-05-17T12:17:51.427109 #1106] DEBUG -- : oo_spawn running cgset -r cpu.cfs_quota_us=30000 /openshift/51965817e6608f0d63000004: {:unsetenv_others=>false, :close_others=>true, :in=>"/dev/null", :out=>#<IO:fd 11>, :err=>#<IO:fd 9>}

At this point no more trace is generated to platform-trace.log

Comment 1 Jhon Honce 2013-05-17 17:56:43 UTC
Cannot reproduce on latest master code.

Comment 2 Clayton Coleman 2013-05-17 20:43:22 UTC
The created app should not be left around if the gear fails (it returns an error to the broker and the user, but doesn't actually roll anything back).

Comment 3 Abhishek Gupta 2013-05-21 20:36:50 UTC
The issue where pending op groups are deleted in case of failures during rollbacks has been fixed and the fix should be in devenv_3257. If we fail to rollback, we will not delete the application. In this case, however, trying to create an application with a cartridge already installed on the node (after changing its bin/control file) resulted in a clean rollback.

Comment 4 Xiaoli Tian 2013-05-22 09:16:37 UTC
Verified it on devenv_3258, making a syntax error in bin/control: line 159

and trying to create app with the url, the syntax error is returned back to client

# rhc app-create ruby2 https://raw.github.com/xltian/rubytestonly/master/metadata/manifest.yml 
The cartridge 'https://raw.github.com/xltian/rubytestonly/master/metadata/manifest.yml' will be downloaded and installed

Application Options
-------------------
  Namespace:  domx1
  Cartridges: https://raw.github.com/xltian/rubytestonly/master/metadata/manifest.yml
  Gear Size:  default
  Scaling:    no

Creating application 'ruby2' ... /var/lib/openshift/57393416c2bf11e2b79912313b12045f/rubytest/bin/control: line 159: syntax error near unexpected token `}'


And the app creation is rolled back, no gears remains in node:

[root@ip-10-202-7-173 ~]# cd /var/lib/openshift/
[root@ip-10-202-7-173 openshift]# ls -lh ruby2
ls: cannot access ruby2: No such file or directory

From platform-trace.log, there's an tear-down operation as well:
D, [2013-05-22T05:10:10.988078 #1087] DEBUG -- : oo_spawn running /sbin/runuser -m -s /bin/sh 57393416c2bf11e2b79912313b12045f -c "exec /usr/bin/runcon 'unconfined_u:system_r:openshift_t:s0:c0,c507' /bin/sh -c \"set -e; /var/lib/openshift/57393416c2bf11e2b79912313b12045f/rubytest/bin/control stop\"": {:unsetenv_others=>true, :close_others=>true, :in=>"/dev/null", :chdir=>"/var/lib/openshift/57393416c2bf11e2b79912313b12045f/", :out=>#<IO:fd 11>, :err=>#<IO:fd 9>}
D, [2013-05-22T05:10:12.058054 #1087] DEBUG -- : oo_spawn buffer(7/) /var/lib/openshift/57393416c2bf11e2b79912313b12045f/rubytest/bin/control: line 159: syntax error near unexpected token `}'

D, [2013-05-22T05:10:12.062315 #1087] DEBUG -- : oo_spawn running cgset -r cpu.cfs_quota_us=30000 /openshift/57393416c2bf11e2b79912313b12045f: {:unsetenv_others=>false, :close_others=>true, :in=>"/dev/null", :out=>#<IO:fd 11>, :err=>#<IO:fd 9>}
D, [2013-05-22T05:10:17.981353 #1087] DEBUG -- : oo_spawn running /sbin/runuser -m -s /bin/sh 57393416c2bf11e2b79912313b12045f -c "exec /usr/bin/runcon 'unconfined_u:system_r:openshift_t:s0:c0,c507' /bin/sh -c \"/var/lib/openshift/57393416c2bf11e2b79912313b12045f/rubytest/bin/teardown\"": {:unsetenv_others=>true, :close_others=>true, :in=>"/dev/null", :chdir=>"/var/lib/openshift/57393416c2bf11e2b79912313b12045f/", :out=>#<IO:fd 11>, :err=>#<IO:fd 9>}
D, [2013-05-22T05:10:21.258854 #1087] DEBUG -- : oo_spawn running /usr/sbin/httxt2dbm -f DB -i /etc/httpd/conf.d/openshift/nodes.txt -o /etc/httpd/conf.d/openshift/nodes.db-20130522-1087-171wqej/new.db: {:unsetenv_others=>false, :close_others=>true, :in=>"/dev/null", :out=>#<IO:fd 14>, :err=>#<IO:fd 11>}
D, [2013-05-22T05:10:21.415206 #1087] DEBUG -- : oo_spawn running service openshift-node-web-proxy reload: {:unsetenv_others=>false, :close_others=>true, :in=>"/dev/null", :out=>#<IO:fd 14>, :err=>#<IO:fd 11>}
D, [2013-05-22T05:10:21.496659 #1087] DEBUG -- : oo_spawn buffer(13/) Reloading node-web-proxy: