Bug 876640

Summary:

oo-admin-move fails when git dir is missing

Product:

OKD

Reporter:

Kenny Woodson <kwoodson>

Component:

Containers

Assignee:

Rob Millner <rmillner>

Status:

CLOSED CURRENTRELEASE

QA Contact:

libra bugs <libra-bugs>

Severity:

medium

Docs Contact:

Priority:

medium

Version:

2.x

CC:

jinzhang, mfisher, qgong

Target Milestone:

---

Target Release:

---

Hardware:

All

OS:

Linux

Whiteboard:

Fixed In Version:

Doc Type:

Bug Fix

Doc Text:

Story Points:

---

Clone Of:

Environment:

Last Closed:

2012-12-19 19:26:49 UTC

Type:

Bug

Regression:

---

Mount Type:

---

Documentation:

---

CRM:

Verified Versions:

Category:

---

oVirt Team:

---

RHEL 7.3 requirements from Atomic Host:

Cloudforms Team:

---

Target Upstream Version:

Embargoed:

Attachments:

Description	Flags
development.log	none

Description Kenny Woodson 2012-11-14 16:16:54 UTC

Description of problem:

DUring some recent moves, I discovered one gear that was missing the git directory but everything else appeared in tact.  The gear was idled.  Env variables were there.  Jenkins dir was there.  app-root was there.  No git directory.

I issue oo-admin-move and the move fails because there was a chcon issued for the git dir and failed.

Version-Release number of selected component (if applicable):
2.0.19.1

How reproducible:

I see it happening currently in prod.  Follow steps below to see if it fails.

Steps to Reproduce:
1. Create a gear
2. Delete the gear's git dir.
3. Idle this gear.
3. Attempt to move this gear.
4. Look at log and verify the chcon fails on the git dir that does not exist.
  
Actual results:
Failed to move this gear.

Expected results:

The gear should move along on its merry way.  We don't care if the gear is broken.  The gear is not running and might have issues.  We want to move it anyway.  

Additional info:
mco.log
I, [2012-11-14T10:51:38.668190 #4165]  INFO -- : openshift.rb:360:in `cartridge_do_action' cartridge_do_action validation = jenkins-1.4 move 'git' 'indam' '
579111b9cdee4c0ebdb816d9f8d92392' '--idle'
I, [2012-11-14T10:51:38.683943 #4165]  INFO -- : openshift.rb:341:in `handle_cartridge_action' handle_cartridge_action executing /usr/bin/runcon -l s0-s0:c0
.c1023 /usr/libexec/openshift/cartridges/jenkins-1.4/info/hooks/move 'git' 'indam' '579111b9cdee4c0ebdb816d9f8d92392' '--idle' 2>&1
I, [2012-11-14T10:51:39.986359 #4165]  INFO -- : openshift.rb:330:in `complete_process_gracefully' ERROR: (1)
------
chcon: cannot access `/var/lib/openshift/579111b9cdee4c0ebdb816d9f8d92392//git/': No such file or directory

------)

gear contents:
[kwoodson ~]$ sudo ls -la /var/lib/openshift/579111b9cdee4c0ebdb816d9f8d92392/
total 76
drwxr-x---.  9 root 579111b9cdee4c0ebdb816d9f8d92392  4096 Nov  6 07:20 .
drwxr-x--x. 11 root root                             36864 Nov 14 10:47 ..
drwxr-xr-x.  4 root 579111b9cdee4c0ebdb816d9f8d92392  4096 Jul  1 04:07 app-root
drwxr-x---.  2 root 579111b9cdee4c0ebdb816d9f8d92392  4096 Jul  1 04:07 .auth
drwxr-x---.  3 root 579111b9cdee4c0ebdb816d9f8d92392  4096 Nov 10 00:13 .env
-rw-r--r--.  1 root root                                56 Jul  1 04:07 .gitconfig
drwxr-xr-x.  7 root root                              4096 Nov  6 07:20 jenkins-1.4
-rw-r--r--.  1 root root                                13 Jul  1 04:07 .jenkins_password
d---------.  3 root root                              4096 Aug 27 17:28 .sandbox
drwxr-x---.  2 root 579111b9cdee4c0ebdb816d9f8d92392  4096 Jul  1 04:07 .ssh
d---------.  3 root root                              4096 Jul  1 04:07 .tmp
[kwoodson ~]$ sudo ls -l /var/lib/openshift/579111b9cdee4c0ebdb816d9f8d92392/.env
total 84
-rw-r--r--. 1 root root  38 Nov  6 07:20 JENKINS_PASSWORD
-rw-r--r--. 1 root root  51 Nov  6 07:20 JENKINS_URL
-rw-r--r--. 1 root root  40 Nov  6 07:20 JENKINS_USERNAME
-rw-r--r--. 1 root root  48 Nov  6 07:20 OPENSHIFT_APP_DNS
-rw-r--r--. 1 root root  31 Nov  6 07:20 OPENSHIFT_APP_NAME
-rw-r--r--. 1 root root  60 Nov  6 07:20 OPENSHIFT_APP_UUID
-rw-r--r--. 1 root root  94 Nov  6 07:20 OPENSHIFT_DATA_DIR
-rw-r--r--. 1 root root  49 Nov  6 07:20 OPENSHIFT_GEAR_DNS
-rw-r--r--. 1 root root  32 Nov  6 07:20 OPENSHIFT_GEAR_NAME
-rw-r--r--. 1 root root  61 Nov  6 07:20 OPENSHIFT_GEAR_UUID
-rw-r--r--. 1 root root  79 Nov  6 07:20 OPENSHIFT_HOMEDIR
-rw-r--r--. 1 root root  43 Nov  6 07:20 OPENSHIFT_INTERNAL_IP
-rw-r--r--. 1 root root  38 Nov  6 07:20 OPENSHIFT_INTERNAL_PORT
-rw-r--r--. 1 root root  41 Nov  6 07:20 OPENSHIFT_JENKINS_IP
-rw-r--r--. 1 root root 104 Nov  6 07:20 OPENSHIFT_JENKINS_LOG_DIR
-rw-r--r--. 1 root root  36 Nov  6 07:20 OPENSHIFT_JENKINS_PORT
-rw-r--r--. 1 root root 102 Nov  6 07:20 OPENSHIFT_REPO_DIR
-rw-r--r--. 1 root root  32 Nov  6 07:20 OPENSHIFT_TMP_DIR
-rw-r--r--. 1 root root 210 Nov  6 07:20 PATH
-rw-r--r--. 1 root root 489 Nov 10 00:13 TYPELESS_TRANSLATED_VARS
-rw-r--r--. 1 root root 284 Nov  6 07:20 USER_VARS

Comment 1 Rob Millner 2012-11-14 18:17:26 UTC

Wrapped the restorecon+chcon in a conditional and non-fatal warning if the target doesn't exist.

https://github.com/openshift/origin-server/pull/902

Comment 2 Rony Gong 🔥 2012-11-15 04:48:39 UTC

Reopened in devenv_2474
Steps:
1.Create php app
2.Then remove the git dir by
rm -rf /var/lib/openshift/d2a5abf019c845a9a23d80089ba1150f/git
3.Move this gear
[root@domU-12-31-38-04-1C-55 openshift]# oo-admin-move --gear_uuid d2a5abf019c845a9a23d80089ba1150f -i domU-12-31-38-04-1C-55

**Notice: C extension not loaded. This is required for optimum MongoDB Ruby driver performance.
  You can install the extension as follows:
  gem install bson_ext

  If you continue to receive this message after installing, make sure that the
  bson_ext gem is in your load path and that the bson_ext and mongo gems are of the same version.

/opt/rh/ruby193/root/usr/share/gems/gems/systemu-1.2.0/lib/systemu.rb:28: Use RbConfig instead of obsolete and deprecated Config.
URL: http://qphp-qgong2.dev.rhcloud.com
Login: qgong
App UUID: d2a5abf019c845a9a23d80089ba1150f
Gear UUID: d2a5abf019c845a9a23d80089ba1150f
DEBUG: Source district uuid: 0f83580d25c24a02b74b58b4e0997e87
DEBUG: Destination district uuid: 0f83580d25c24a02b74b58b4e0997e87
DEBUG: District unchanged keeping uid
DEBUG: Getting existing app 'qphp' status before moving
DEBUG: Gear component 'php-5.3' was running
DEBUG: Stopping existing app cartridge 'php-5.3' before moving
DEBUG: Force stopping existing app cartridge 'php-5.3' before moving
DEBUG: Creating new account for gear 'qphp' on domU-12-31-38-04-1C-55
DEBUG: Moving content for app 'qphp', gear 'qphp' to domU-12-31-38-04-1C-55
Identity added: /var/www/openshift/broker/config/keys/rsync_id_rsa (/var/www/openshift/broker/config/keys/rsync_id_rsa)
Warning: Permanently added '10.220.31.163' (RSA) to the list of known hosts.
Agent pid 7189
unset SSH_AUTH_SOCK;
unset SSH_AGENT_PID;
echo Agent pid 7189 killed;
DEBUG: Performing cartridge level move for 'php-5.3' on domU-12-31-38-04-1C-55
DEBUG: Moving failed.  Rolling back gear 'qphp' 'qphp' with remove-httpd-proxy on 'domU-12-31-38-04-1C-55'
DEBUG: Moving failed.  Rolling back gear 'qphp' in 'qphp' with destroy on 'domU-12-31-38-04-1C-55'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.1.2/lib/openshift/mcollective_application_container_proxy.rb:1265:in `run_cartridge_command': Node execution failure (invalid exit code from node).  If the problem persists please contact Red Hat support. (OpenShift::NodeException)
	from /var/www/openshift/broker/lib/express/broker/mcollective_ext.rb:12:in `run_cartridge_command'
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.1.2/lib/openshift/mcollective_application_container_proxy.rb:802:in `block in move_gear'
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.1.2/lib/openshift/mcollective_application_container_proxy.rb:776:in `each'
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.1.2/lib/openshift/mcollective_application_container_proxy.rb:776:in `move_gear'
	from /usr/sbin/oo-admin-move:111:in `<main>'

Comment 3 Rony Gong 🔥 2012-11-15 04:48:58 UTC

Created attachment 645334 [details]
development.log

Comment 4 Rob Millner 2012-11-15 22:40:59 UTC

We ended up having one additional issue.  Resolved in pull request:
https://github.com/openshift/origin-server/pull/924

[root@ip-10-85-155-81 ~]# ls /var/lib/openshift/3a0cd4dbca4a458aa35f88500c09b079
app-root  diy-0.1


[root@ip-10-85-155-81 ~]# oo-admin-move --gear_uuid 3a0cd4dbca4a458aa35f88500c09b079

**Notice: C extension not loaded. This is required for optimum MongoDB Ruby driver performance.
  You can install the extension as follows:
  gem install bson_ext

  If you continue to receive this message after installing, make sure that the
  bson_ext gem is in your load path and that the bson_ext and mongo gems are of the same version.

Mocha deprecation warning: Test::Unit or MiniTest must be loaded *before* Mocha.
Mocha deprecation warning: If you're integrating with another test library, you should probably require 'mocha_standalone' instead of 'mocha'
/opt/rh/ruby193/root/usr/share/gems/gems/systemu-1.2.0/lib/systemu.rb:28: Use RbConfig instead of obsolete and deprecated Config.
URL: http://rmtest-rmillner206.dev.rhcloud.com
Login: rmillner
App UUID: 3a0cd4dbca4a458aa35f88500c09b079
Gear UUID: 3a0cd4dbca4a458aa35f88500c09b079
DEBUG: Destination container: ip-10-85-155-81
DEBUG: Source district uuid: NONE
DEBUG: Destination district uuid: NONE
DEBUG: Getting existing app 'rmtest' status before moving
DEBUG: Gear component 'diy-0.1' was running
DEBUG: Stopping existing app cartridge 'diy-0.1' before moving
DEBUG: Force stopping existing app cartridge 'diy-0.1' before moving
DEBUG: Reserved uid '' on district: 'NONE'
DEBUG: Creating new account for gear 'rmtest' on ip-10-85-155-81
DEBUG: Moving content for app 'rmtest', gear 'rmtest' to ip-10-85-155-81
Identity added: /var/www/openshift/broker/config/keys/rsync_id_rsa (/var/www/openshift/broker/config/keys/rsync_id_rsa)
Warning: Permanently added '10.84.231.130' (RSA) to the list of known hosts.
Warning: Permanently added '10.85.155.81' (RSA) to the list of known hosts.
Agent pid 16261
unset SSH_AUTH_SOCK;
unset SSH_AGENT_PID;
echo Agent pid 16261 killed;
DEBUG: Performing cartridge level move for 'diy-0.1' on ip-10-85-155-81
DEBUG: Starting cartridge 'diy-0.1' in 'rmtest' after move on ip-10-85-155-81
DEBUG: Fixing DNS and mongo for gear 'rmtest' after move
DEBUG: Changing server identity of 'rmtest' from 'ip-10-84-231-130' to 'ip-10-85-155-81'
DEBUG: Deconfiguring old app 'rmtest' on ip-10-84-231-130 after move
Successfully moved 'rmtest' with gear uuid '3a0cd4dbca4a458aa35f88500c09b079' from 'ip-10-84-231-130' to 'ip-10-85-155-81'

Comment 5 Rony Gong 🔥 2012-11-16 10:36:37 UTC

Verified on devenv_2488
1.remove the git dir
2.move
[root@domU-12-31-38-04-28-01 openshift]# oo-admin-move --gear_uuid fa660fabcd61459fa2ba1b2792dce55b -i ip-10-122-61-248
URL: http://qzend-qgong9.dev.rhcloud.com
Login: qgong
App UUID: fa660fabcd61459fa2ba1b2792dce55b
Gear UUID: fa660fabcd61459fa2ba1b2792dce55b
DEBUG: Source district uuid: 846b9f9f1fa3484996d22721ff73e5b2
DEBUG: Destination district uuid: 846b9f9f1fa3484996d22721ff73e5b2
DEBUG: District unchanged keeping uid
DEBUG: Getting existing app 'qzend' status before moving
DEBUG: Gear component 'zend-5.6' was running
DEBUG: Stopping existing app cartridge 'zend-5.6' before moving
DEBUG: Force stopping existing app cartridge 'zend-5.6' before moving
DEBUG: Creating new account for gear 'qzend' on ip-10-122-61-248
DEBUG: Moving content for app 'qzend', gear 'qzend' to ip-10-122-61-248
Identity added: /var/www/openshift/broker/config/keys/rsync_id_rsa (/var/www/openshift/broker/config/keys/rsync_id_rsa)
Warning: Permanently added '10.220.43.239' (RSA) to the list of known hosts.
Warning: Permanently added '10.122.61.248' (RSA) to the list of known hosts.
Agent pid 12026
unset SSH_AUTH_SOCK;
unset SSH_AGENT_PID;
echo Agent pid 12026 killed;
DEBUG: Performing cartridge level move for 'zend-5.6' on ip-10-122-61-248
DEBUG: Starting cartridge 'zend-5.6' in 'qzend' after move on ip-10-122-61-248
DEBUG: Fixing DNS and mongo for gear 'qzend' after move
DEBUG: Changing server identity of 'qzend' from 'domU-12-31-38-04-28-01' to 'ip-10-122-61-248'
DEBUG: Deconfiguring old app 'qzend' on domU-12-31-38-04-28-01 after move
Successfully moved 'qzend' with gear uuid 'fa660fabcd61459fa2ba1b2792dce55b' from 'domU-12-31-38-04-28-01' to 'ip-10-122-61-248'


development.log
2012-11-16 05:28:06.179 [DEBUG] DEBUG: [#<MCollective::RPC::Result:0x0000000519e240 @agent="openshift", @action="cartridge_do", @results={:sender=>"ip-10-122-61-248", :statuscode=>0, :statusmsg=>"OK", :data=>{:exitcode=>0, :output=>"The dir does not exist: /var/lib/openshift/fa660fabcd61459fa2ba1b2792dce55b//git/\nWARNING: No git repo; cannot redeploy.\nFILES_DIR=/usr/libexec/openshift/cartridges/zend-5.6/files/user-files CART_DIR=/var/lib/openshift/fa660fabcd61459fa2ba1b2792dce55b//zend-5.6 USER_ID=1000 GROUP_ID=1000 IP_ADDRESS=127.1.244.1\nConfiguring zend server on move\n"}}>] (pid:11606)
2012-11-16 05:28:06.182 [DEBUG] DEBUG: Cartridge command zend-5.6::move exitcode = 0 (pid:11606)