Hide Forgot
Description of problem: Upgrade server from sprint34 to latest devenv, after oo-admin-upgrade, there are some broken symlinks under gear's home dir, and the app cannot be visited. [525e366bb39175593d0000e5] # find . -type l|xargs file |grep broken ./app-root/dependencies: broken symbolic link to `/var/lib/openshift/525e366bb39175593d0000e5/app-root/runtime/dependencies' ./app-root/repo: broken symbolic link to `/var/lib/openshift/525e366bb39175593d0000e5/app-root/runtime/repo' ./app-root/build-dependencies: broken symbolic link to `/var/lib/openshift/525e366bb39175593d0000e5/app-root/runtime/build-dependencies' Version-Release number of selected component (if applicable): devenv_3901 How reproducible: always Steps to Reproduce: 1. Create app on devenv-stage_496 2. Upgrade the server and migrate the datastore 3. Upgrade gears with oo-admin-upgrade #oo-admin-upgrade upgrade-node --version 2.0.35 --ignore-cartridge-version 4. Check the gear home dir after upgrade Actual results: There are some broken symlinks for the gear and the app cannot work well. Expected results: The app should work well and the symlinks point to the correct path. Additional info: After deploy build from client side, the app can works well and the symlinks point to the right path. But new broken symlink generated under app-deployments dir: [app-deployments]# find -L . -type l|xargs file|grep -i broken ./app-deployments/by-id/e41e277e: broken symbolic link to `../2013-10-16_04-16-19.800' [app-deployments]# tree . ├── 2013-10-16_05-31-10.144 │ ├── build-dependencies │ ├── dependencies │ ├── metadata.json │ └── repo │ ├── 321 │ ├── config.ru │ ├── public │ ├── README.md │ └── tmp └── by-id ├── e41e277e -> ../2013-10-16_04-16-19.800 └── f6c45c58 -> ../2013-10-16_05-31-10.144
https://github.com/openshift/li/pull/2016
Commit pushed to master at https://github.com/openshift/li https://github.com/openshift/li/commit/2c0e1b324ff74b46e7371bb646b35c458c71478b Bug 1019732: Make deployment upgrade steps idempotent
Checked on devenv_3927, the issue still can be reproduced. Assign the bug back.
Meng, I've been unable to reproduce this issue when upgrading devenv-stage_521 => devenv_3928 using mock/php/python scalable and ruby non-scalable applications. Can you try again with the latest stage image, and if you're able to reproduce it, give me the stage/dev image names and also the specifics of the applications upgraded (cartridge type, scaling options, etc)? Thanks!
Hi Dan, I can reproduce this issue from devenv-stage_528 to devenv_3932 for all cartridges. After migration, all the gears home dir has broken symlinks. Take a php app as example, [root@ip-10-180-180-11 wordpress-bmengmdev]# find . -type l|xargs file |grep broken ./app-root/dependencies: broken symbolic link to `/var/lib/openshift/52675eb9c7baf242b900035c/app-root/runtime/dependencies' ./app-root/repo: broken symbolic link to `/var/lib/openshift/52675eb9c7baf242b900035c/app-root/runtime/repo' ./app-root/build-dependencies: broken symbolic link to `/var/lib/openshift/52675eb9c7baf242b900035c/app-root/runtime/build-dependencies' Check the runtime dir under app-root, [root@ip-10-180-180-11 runtime]# ls -al total 16 drwxr-x---. 3 52675eb9c7baf242b900035c 52675eb9c7baf242b900035c 4096 Oct 23 02:06 . drwxr-xr-x. 4 root 52675eb9c7baf242b900035c 4096 Oct 23 02:05 .. lrwxrwxrwx. 1 52675eb9c7baf242b900035c 52675eb9c7baf242b900035c 7 Oct 23 01:29 data -> ../data drwxr-x---. 3 52675eb9c7baf242b900035c 52675eb9c7baf242b900035c 4096 Oct 23 02:05 repo.backup -rw-r-----. 1 52675eb9c7baf242b900035c 52675eb9c7baf242b900035c 8 Oct 23 02:06 .state Create a new app on the same instance and check the runtime dir: [root@ip-10-180-180-11 runtime]# ls -al total 24 drwxr-x---. 2 52676bafc7baf290a8000019 52676bafc7baf290a8000019 4096 Oct 23 02:25 . drwxr-xr-x. 4 root 52676bafc7baf290a8000019 4096 Oct 23 02:24 .. lrwxrwxrwx. 1 52676bafc7baf290a8000019 52676bafc7baf290a8000019 102 Oct 23 02:24 build-dependencies -> /var/lib/openshift/52676bafc7baf290a8000019/app-deployments/2013-10-23_02-24-52.738/build-dependencies lrwxrwxrwx. 1 52676bafc7baf290a8000019 52676bafc7baf290a8000019 7 Oct 23 02:24 data -> ../data lrwxrwxrwx. 1 52676bafc7baf290a8000019 52676bafc7baf290a8000019 96 Oct 23 02:24 dependencies -> /var/lib/openshift/52676bafc7baf290a8000019/app-deployments/2013-10-23_02-24-52.738/dependencies lrwxrwxrwx. 1 52676bafc7baf290a8000019 52676bafc7baf290a8000019 88 Oct 23 02:25 repo -> /var/lib/openshift/52676bafc7baf290a8000019/app-deployments/2013-10-23_02-24-52.738/repo -rw-r-----. 1 52676bafc7baf290a8000019 52676bafc7baf290a8000019 8 Oct 23 02:25 .state
These directories are now physical rather than symlinks as of some redesign work which hit master last night: https://github.com/openshift/li/pull/2025 Please re-test with devenv_3934- there shouldn't be any links at all anymore. Sorry for the back-and-forth, Meng, and thanks for your patience.
Checked on devenv_3939, after migration, there is no broken symlinks left. [root@ip-10-138-22-172 spring-bmengmdev2]# find . -type l|xargs file |grep broken [root@ip-10-138-22-172 spring-bmengmdev2]# And the directories are created as physical dir. [root@ip-10-138-22-172 spring-bmengmdev2]# ls -l app-root/runtime/ total 16 drwxr-xr-x. 2 root root 4096 Oct 24 06:00 build-dependencies lrwxrwxrwx. 1 5268e390631abb84c6000059 5268e390631abb84c6000059 7 Oct 24 05:08 data -> ../data drwxr-xr-x. 2 root root 4096 Oct 24 06:00 dependencies -rw-r--r--. 1 5268e390631abb84c6000059 5268e390631abb84c6000059 6 Oct 24 06:01 jbosseap.pid drwxr-x---. 6 5268e390631abb84c6000059 5268e390631abb84c6000059 4096 Oct 24 05:20 repo App can be visited after gear upgrade. Move bug to verified.