Description of problem: DUring some recent moves, I discovered one gear that was missing the git directory but everything else appeared in tact. The gear was idled. Env variables were there. Jenkins dir was there. app-root was there. No git directory. I issue oo-admin-move and the move fails because there was a chcon issued for the git dir and failed. Version-Release number of selected component (if applicable): 2.0.19.1 How reproducible: I see it happening currently in prod. Follow steps below to see if it fails. Steps to Reproduce: 1. Create a gear 2. Delete the gear's git dir. 3. Idle this gear. 3. Attempt to move this gear. 4. Look at log and verify the chcon fails on the git dir that does not exist. Actual results: Failed to move this gear. Expected results: The gear should move along on its merry way. We don't care if the gear is broken. The gear is not running and might have issues. We want to move it anyway. Additional info: mco.log I, [2012-11-14T10:51:38.668190 #4165] INFO -- : openshift.rb:360:in `cartridge_do_action' cartridge_do_action validation = jenkins-1.4 move 'git' 'indam' ' 579111b9cdee4c0ebdb816d9f8d92392' '--idle' I, [2012-11-14T10:51:38.683943 #4165] INFO -- : openshift.rb:341:in `handle_cartridge_action' handle_cartridge_action executing /usr/bin/runcon -l s0-s0:c0 .c1023 /usr/libexec/openshift/cartridges/jenkins-1.4/info/hooks/move 'git' 'indam' '579111b9cdee4c0ebdb816d9f8d92392' '--idle' 2>&1 I, [2012-11-14T10:51:39.986359 #4165] INFO -- : openshift.rb:330:in `complete_process_gracefully' ERROR: (1) ------ chcon: cannot access `/var/lib/openshift/579111b9cdee4c0ebdb816d9f8d92392//git/': No such file or directory ------) gear contents: [kwoodson ~]$ sudo ls -la /var/lib/openshift/579111b9cdee4c0ebdb816d9f8d92392/ total 76 drwxr-x---. 9 root 579111b9cdee4c0ebdb816d9f8d92392 4096 Nov 6 07:20 . drwxr-x--x. 11 root root 36864 Nov 14 10:47 .. drwxr-xr-x. 4 root 579111b9cdee4c0ebdb816d9f8d92392 4096 Jul 1 04:07 app-root drwxr-x---. 2 root 579111b9cdee4c0ebdb816d9f8d92392 4096 Jul 1 04:07 .auth drwxr-x---. 3 root 579111b9cdee4c0ebdb816d9f8d92392 4096 Nov 10 00:13 .env -rw-r--r--. 1 root root 56 Jul 1 04:07 .gitconfig drwxr-xr-x. 7 root root 4096 Nov 6 07:20 jenkins-1.4 -rw-r--r--. 1 root root 13 Jul 1 04:07 .jenkins_password d---------. 3 root root 4096 Aug 27 17:28 .sandbox drwxr-x---. 2 root 579111b9cdee4c0ebdb816d9f8d92392 4096 Jul 1 04:07 .ssh d---------. 3 root root 4096 Jul 1 04:07 .tmp [kwoodson ~]$ sudo ls -l /var/lib/openshift/579111b9cdee4c0ebdb816d9f8d92392/.env total 84 -rw-r--r--. 1 root root 38 Nov 6 07:20 JENKINS_PASSWORD -rw-r--r--. 1 root root 51 Nov 6 07:20 JENKINS_URL -rw-r--r--. 1 root root 40 Nov 6 07:20 JENKINS_USERNAME -rw-r--r--. 1 root root 48 Nov 6 07:20 OPENSHIFT_APP_DNS -rw-r--r--. 1 root root 31 Nov 6 07:20 OPENSHIFT_APP_NAME -rw-r--r--. 1 root root 60 Nov 6 07:20 OPENSHIFT_APP_UUID -rw-r--r--. 1 root root 94 Nov 6 07:20 OPENSHIFT_DATA_DIR -rw-r--r--. 1 root root 49 Nov 6 07:20 OPENSHIFT_GEAR_DNS -rw-r--r--. 1 root root 32 Nov 6 07:20 OPENSHIFT_GEAR_NAME -rw-r--r--. 1 root root 61 Nov 6 07:20 OPENSHIFT_GEAR_UUID -rw-r--r--. 1 root root 79 Nov 6 07:20 OPENSHIFT_HOMEDIR -rw-r--r--. 1 root root 43 Nov 6 07:20 OPENSHIFT_INTERNAL_IP -rw-r--r--. 1 root root 38 Nov 6 07:20 OPENSHIFT_INTERNAL_PORT -rw-r--r--. 1 root root 41 Nov 6 07:20 OPENSHIFT_JENKINS_IP -rw-r--r--. 1 root root 104 Nov 6 07:20 OPENSHIFT_JENKINS_LOG_DIR -rw-r--r--. 1 root root 36 Nov 6 07:20 OPENSHIFT_JENKINS_PORT -rw-r--r--. 1 root root 102 Nov 6 07:20 OPENSHIFT_REPO_DIR -rw-r--r--. 1 root root 32 Nov 6 07:20 OPENSHIFT_TMP_DIR -rw-r--r--. 1 root root 210 Nov 6 07:20 PATH -rw-r--r--. 1 root root 489 Nov 10 00:13 TYPELESS_TRANSLATED_VARS -rw-r--r--. 1 root root 284 Nov 6 07:20 USER_VARS
Wrapped the restorecon+chcon in a conditional and non-fatal warning if the target doesn't exist. https://github.com/openshift/origin-server/pull/902
Reopened in devenv_2474 Steps: 1.Create php app 2.Then remove the git dir by rm -rf /var/lib/openshift/d2a5abf019c845a9a23d80089ba1150f/git 3.Move this gear [root@domU-12-31-38-04-1C-55 openshift]# oo-admin-move --gear_uuid d2a5abf019c845a9a23d80089ba1150f -i domU-12-31-38-04-1C-55 **Notice: C extension not loaded. This is required for optimum MongoDB Ruby driver performance. You can install the extension as follows: gem install bson_ext If you continue to receive this message after installing, make sure that the bson_ext gem is in your load path and that the bson_ext and mongo gems are of the same version. /opt/rh/ruby193/root/usr/share/gems/gems/systemu-1.2.0/lib/systemu.rb:28: Use RbConfig instead of obsolete and deprecated Config. URL: http://qphp-qgong2.dev.rhcloud.com Login: qgong App UUID: d2a5abf019c845a9a23d80089ba1150f Gear UUID: d2a5abf019c845a9a23d80089ba1150f DEBUG: Source district uuid: 0f83580d25c24a02b74b58b4e0997e87 DEBUG: Destination district uuid: 0f83580d25c24a02b74b58b4e0997e87 DEBUG: District unchanged keeping uid DEBUG: Getting existing app 'qphp' status before moving DEBUG: Gear component 'php-5.3' was running DEBUG: Stopping existing app cartridge 'php-5.3' before moving DEBUG: Force stopping existing app cartridge 'php-5.3' before moving DEBUG: Creating new account for gear 'qphp' on domU-12-31-38-04-1C-55 DEBUG: Moving content for app 'qphp', gear 'qphp' to domU-12-31-38-04-1C-55 Identity added: /var/www/openshift/broker/config/keys/rsync_id_rsa (/var/www/openshift/broker/config/keys/rsync_id_rsa) Warning: Permanently added '10.220.31.163' (RSA) to the list of known hosts. Agent pid 7189 unset SSH_AUTH_SOCK; unset SSH_AGENT_PID; echo Agent pid 7189 killed; DEBUG: Performing cartridge level move for 'php-5.3' on domU-12-31-38-04-1C-55 DEBUG: Moving failed. Rolling back gear 'qphp' 'qphp' with remove-httpd-proxy on 'domU-12-31-38-04-1C-55' DEBUG: Moving failed. Rolling back gear 'qphp' in 'qphp' with destroy on 'domU-12-31-38-04-1C-55' /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.1.2/lib/openshift/mcollective_application_container_proxy.rb:1265:in `run_cartridge_command': Node execution failure (invalid exit code from node). If the problem persists please contact Red Hat support. (OpenShift::NodeException) from /var/www/openshift/broker/lib/express/broker/mcollective_ext.rb:12:in `run_cartridge_command' from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.1.2/lib/openshift/mcollective_application_container_proxy.rb:802:in `block in move_gear' from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.1.2/lib/openshift/mcollective_application_container_proxy.rb:776:in `each' from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.1.2/lib/openshift/mcollective_application_container_proxy.rb:776:in `move_gear' from /usr/sbin/oo-admin-move:111:in `<main>'
Created attachment 645334 [details] development.log
We ended up having one additional issue. Resolved in pull request: https://github.com/openshift/origin-server/pull/924 [root@ip-10-85-155-81 ~]# ls /var/lib/openshift/3a0cd4dbca4a458aa35f88500c09b079 app-root diy-0.1 [root@ip-10-85-155-81 ~]# oo-admin-move --gear_uuid 3a0cd4dbca4a458aa35f88500c09b079 **Notice: C extension not loaded. This is required for optimum MongoDB Ruby driver performance. You can install the extension as follows: gem install bson_ext If you continue to receive this message after installing, make sure that the bson_ext gem is in your load path and that the bson_ext and mongo gems are of the same version. Mocha deprecation warning: Test::Unit or MiniTest must be loaded *before* Mocha. Mocha deprecation warning: If you're integrating with another test library, you should probably require 'mocha_standalone' instead of 'mocha' /opt/rh/ruby193/root/usr/share/gems/gems/systemu-1.2.0/lib/systemu.rb:28: Use RbConfig instead of obsolete and deprecated Config. URL: http://rmtest-rmillner206.dev.rhcloud.com Login: rmillner App UUID: 3a0cd4dbca4a458aa35f88500c09b079 Gear UUID: 3a0cd4dbca4a458aa35f88500c09b079 DEBUG: Destination container: ip-10-85-155-81 DEBUG: Source district uuid: NONE DEBUG: Destination district uuid: NONE DEBUG: Getting existing app 'rmtest' status before moving DEBUG: Gear component 'diy-0.1' was running DEBUG: Stopping existing app cartridge 'diy-0.1' before moving DEBUG: Force stopping existing app cartridge 'diy-0.1' before moving DEBUG: Reserved uid '' on district: 'NONE' DEBUG: Creating new account for gear 'rmtest' on ip-10-85-155-81 DEBUG: Moving content for app 'rmtest', gear 'rmtest' to ip-10-85-155-81 Identity added: /var/www/openshift/broker/config/keys/rsync_id_rsa (/var/www/openshift/broker/config/keys/rsync_id_rsa) Warning: Permanently added '10.84.231.130' (RSA) to the list of known hosts. Warning: Permanently added '10.85.155.81' (RSA) to the list of known hosts. Agent pid 16261 unset SSH_AUTH_SOCK; unset SSH_AGENT_PID; echo Agent pid 16261 killed; DEBUG: Performing cartridge level move for 'diy-0.1' on ip-10-85-155-81 DEBUG: Starting cartridge 'diy-0.1' in 'rmtest' after move on ip-10-85-155-81 DEBUG: Fixing DNS and mongo for gear 'rmtest' after move DEBUG: Changing server identity of 'rmtest' from 'ip-10-84-231-130' to 'ip-10-85-155-81' DEBUG: Deconfiguring old app 'rmtest' on ip-10-84-231-130 after move Successfully moved 'rmtest' with gear uuid '3a0cd4dbca4a458aa35f88500c09b079' from 'ip-10-84-231-130' to 'ip-10-85-155-81'
Verified on devenv_2488 1.remove the git dir 2.move [root@domU-12-31-38-04-28-01 openshift]# oo-admin-move --gear_uuid fa660fabcd61459fa2ba1b2792dce55b -i ip-10-122-61-248 URL: http://qzend-qgong9.dev.rhcloud.com Login: qgong App UUID: fa660fabcd61459fa2ba1b2792dce55b Gear UUID: fa660fabcd61459fa2ba1b2792dce55b DEBUG: Source district uuid: 846b9f9f1fa3484996d22721ff73e5b2 DEBUG: Destination district uuid: 846b9f9f1fa3484996d22721ff73e5b2 DEBUG: District unchanged keeping uid DEBUG: Getting existing app 'qzend' status before moving DEBUG: Gear component 'zend-5.6' was running DEBUG: Stopping existing app cartridge 'zend-5.6' before moving DEBUG: Force stopping existing app cartridge 'zend-5.6' before moving DEBUG: Creating new account for gear 'qzend' on ip-10-122-61-248 DEBUG: Moving content for app 'qzend', gear 'qzend' to ip-10-122-61-248 Identity added: /var/www/openshift/broker/config/keys/rsync_id_rsa (/var/www/openshift/broker/config/keys/rsync_id_rsa) Warning: Permanently added '10.220.43.239' (RSA) to the list of known hosts. Warning: Permanently added '10.122.61.248' (RSA) to the list of known hosts. Agent pid 12026 unset SSH_AUTH_SOCK; unset SSH_AGENT_PID; echo Agent pid 12026 killed; DEBUG: Performing cartridge level move for 'zend-5.6' on ip-10-122-61-248 DEBUG: Starting cartridge 'zend-5.6' in 'qzend' after move on ip-10-122-61-248 DEBUG: Fixing DNS and mongo for gear 'qzend' after move DEBUG: Changing server identity of 'qzend' from 'domU-12-31-38-04-28-01' to 'ip-10-122-61-248' DEBUG: Deconfiguring old app 'qzend' on domU-12-31-38-04-28-01 after move Successfully moved 'qzend' with gear uuid 'fa660fabcd61459fa2ba1b2792dce55b' from 'domU-12-31-38-04-28-01' to 'ip-10-122-61-248' development.log 2012-11-16 05:28:06.179 [DEBUG] DEBUG: [#<MCollective::RPC::Result:0x0000000519e240 @agent="openshift", @action="cartridge_do", @results={:sender=>"ip-10-122-61-248", :statuscode=>0, :statusmsg=>"OK", :data=>{:exitcode=>0, :output=>"The dir does not exist: /var/lib/openshift/fa660fabcd61459fa2ba1b2792dce55b//git/\nWARNING: No git repo; cannot redeploy.\nFILES_DIR=/usr/libexec/openshift/cartridges/zend-5.6/files/user-files CART_DIR=/var/lib/openshift/fa660fabcd61459fa2ba1b2792dce55b//zend-5.6 USER_ID=1000 GROUP_ID=1000 IP_ADDRESS=127.1.244.1\nConfiguring zend server on move\n"}}>] (pid:11606) 2012-11-16 05:28:06.182 [DEBUG] DEBUG: Cartridge command zend-5.6::move exitcode = 0 (pid:11606)