Bug 998926

Summary: [origin_runtime_156]Failed to switch to Apache/mod_wsgi for existing python-2.7 app after migration
Product: OpenShift Online Reporter: chunchen <chunchen>
Component: ContainersAssignee: Rob Millner <rmillner>
Status: CLOSED CURRENTRELEASE QA Contact: libra bugs <libra-bugs>
Severity: medium Docs Contact:
Priority: medium    
Version: 2.xCC: mfisher, wsun
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-08-29 12:53:47 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description chunchen 2013-08-20 11:04:39 UTC
Description of problem:
When disable app.py after migration,it is failed to use Apache/mod_wsgi for python-2.7 app and met "cannot open shared object file" errors, the "python app.py" process also did not startup.

Version-Release number of selected component (if applicable):
devenv_stage_442 to devenv_3678

How reproducible:
always

Steps to Reproduce:
1. Create a python-2.7 app
rhc app create cpy27s3mingears python-2.7 --enable-jenkins
2. Do upgrade and migrate
3. Disable app.py and git push
cd cpy27s3mingears
mv app.py app.py.disabled
git add .; git commit -amp; git push
4. Check this app's process
rhc ssh cpy27s3mingears 'ps -ef'

Actual results:
At step 3:
<--------------snip--------------->
Warning : There are multiple branch changesets here
[repo] $ /bin/sh -xe /tmp/hudson8167012588286808101.sh
+ alias 'rsync=rsync --exclude-from='\''/var/lib/openshift/434749237534258521702400/python//metadata/rsync.excludes'\'' --delete-after -az -e '\''/usr/libexec/openshift/cartridges/jenkins/bin/git_ssh_wrapper.sh'\'''
+ upstream_ssh=50592144097b11e3a62922000a9ab0c2.rhcloud.com
+ rsync --exclude-from=/var/lib/openshift/434749237534258521702400/python//metadata/rsync.excludes --delete-after -az -e /usr/libexec/openshift/cartridges/jenkins/bin/git_ssh_wrapper.sh '50592144097b11e3a62922000a9ab0c2.rhcloud.com:${OPENSHIFT_PYTHON_DIR}virtenv/' /var/lib/openshift/434749237534258521702400/python/virtenv/
+ gear build
running develop
running egg_info
writing requirements to YourAppName.egg-info/requires.txt
writing YourAppName.egg-info/PKG-INFO
writing top-level names to YourAppName.egg-info/top_level.txt
writing dependency_links to YourAppName.egg-info/dependency_links.txt
writing requirements to YourAppName.egg-info/requires.txt
writing YourAppName.egg-info/PKG-INFO
writing top-level names to YourAppName.egg-info/top_level.txt
writing dependency_links to YourAppName.egg-info/dependency_links.txt
warning: manifest_maker: standard file 'setup.py' not found

reading manifest file 'YourAppName.egg-info/SOURCES.txt'
writing manifest file 'YourAppName.egg-info/SOURCES.txt'
running build_ext
Creating /var/lib/openshift/434749237534258521702400/python/virtenv/lib/python2.7/site-packages/YourAppName.egg-link (link to .)
YourAppName 1.0 is already the active version in easy-install.pth

Installed /var/lib/openshift/434749237534258521702400/python/virtenv
Processing dependencies for YourAppName==1.0
Searching for gevent==0.13.8
Best match: gevent 0.13.8
Processing gevent-0.13.8-py2.7-linux-x86_64.egg
gevent 0.13.8 is already the active version in easy-install.pth

Using /var/lib/openshift/434749237534258521702400/python/virtenv/lib/python2.7/site-packages/gevent-0.13.8-py2.7-linux-x86_64.egg
Searching for greenlet==0.4.1
Best match: greenlet 0.4.1
Processing greenlet-0.4.1-py2.7-linux-x86_64.egg
greenlet 0.4.1 is already the active version in easy-install.pth

Using /var/lib/openshift/434749237534258521702400/python/virtenv/lib/python2.7/site-packages/greenlet-0.4.1-py2.7-linux-x86_64.egg
Finished processing dependencies for YourAppName==1.0
Script /var/lib/openshift/434749237534258521702400/python//virtenv/bin/activate.csh cannot be made relative (it's not a normal script that starts with #!/var/lib/openshift/434749237534258521702400/python/virtenv/bin/python)
Script /var/lib/openshift/434749237534258521702400/python//virtenv/bin/activate.fish cannot be made relative (it's not a normal script that starts with #!/var/lib/openshift/434749237534258521702400/python/virtenv/bin/python)
+ /usr/libexec/openshift/cartridges/jenkins/bin/git_ssh_wrapper.sh 50592144097b11e3a62922000a9ab0c2.rhcloud.com 'gear stop'
Stopping gear...
Stopping PYTHON cart
+ rsync --exclude-from=/var/lib/openshift/434749237534258521702400/python//metadata/rsync.excludes --delete-after -az -e /usr/libexec/openshift/cartridges/jenkins/bin/git_ssh_wrapper.sh /var/lib/openshift/434749237534258521702400/python/virtenv/ '50592144097b11e3a62922000a9ab0c2.rhcloud.com:${OPENSHIFT_PYTHON_DIR}virtenv/'
+ rsync --exclude-from=/var/lib/openshift/434749237534258521702400/python//metadata/rsync.excludes --delete-after -az -e /usr/libexec/openshift/cartridges/jenkins/bin/git_ssh_wrapper.sh /var/lib/openshift/434749237534258521702400/app-root/runtime/repo/ '50592144097b11e3a62922000a9ab0c2.rhcloud.com:${OPENSHIFT_REPO_DIR}'
+ /usr/libexec/openshift/cartridges/jenkins/bin/git_ssh_wrapper.sh 50592144097b11e3a62922000a9ab0c2.rhcloud.com 'gear remotedeploy'
Starting application cpy27s3mingears
Script /var/lib/openshift/50592144097b11e3a62922000a9ab0c2/python//virtenv/bin/activate.csh cannot be made relative (it's not a normal script that starts with #!/var/lib/openshift/50592144097b11e3a62922000a9ab0c2/python/virtenv/bin/python)
Script /var/lib/openshift/50592144097b11e3a62922000a9ab0c2/python//virtenv/bin/activate.fish cannot be made relative (it's not a normal script that starts with #!/var/lib/openshift/50592144097b11e3a62922000a9ab0c2/python/virtenv/bin/python)
Starting PYTHON cart
httpd: Syntax error on line 7 of /var/lib/openshift/50592144097b11e3a62922000a9ab0c2/python/etc/conf/httpd_nolog.conf: Cannot load /var/lib/openshift/50592144097b11e3a62922000a9ab0c2/python/modules/mod_version.so into server: /var/lib/openshift/50592144097b11e3a62922000a9ab0c2/python/modules/mod_version.so: cannot open shared object file: No such file or directory
An error occurred executing 'gear remotedeploy' (exit code: 1)
Error message: Failed to execute: 'control start' for /var/lib/openshift/50592144097b11e3a62922000a9ab0c2/python

For more details about the problem, try running the command again with the '--trace' option.
Build step 'Execute shell' marked build as failure
Archiving artifacts
Finished: FAILURE

At step 4:
UID        PID  PPID  C STIME TTY          TIME CMD
501      21962 21940  0 06:56 ?        00:00:00 sshd: 50592144097b11e3a62922000a9ab0c2@notty
501      21963 21962  4 06:56 ?        00:00:00 ps -ef

Expected results:
Python-2.7 app should can switch to Apache/mod_wsgi after migration.

Additional info:

Comment 1 Rob Millner 2013-08-20 19:01:51 UTC
Pull request:
https://github.com/openshift/origin-server/pull/3438

Comment 2 openshift-github-bot 2013-08-20 21:40:28 UTC
Commit pushed to master at https://github.com/openshift/origin-server

https://github.com/openshift/origin-server/commit/76e3715f56335443f871efbf25177ff40f6fbded
Bug 998926 - setup needs a reentrant version of creating the Apache symlinks

Comment 3 chunchen 2013-08-22 05:54:07 UTC
It's fixed, verified on devenv_stage_442 to devenv_stage_448, please refer to the following results:

1. Create a python-2.7 app
2. Do upgrade and migrate
3. Disable app.py and git push
# git add .; git commit -amp; git push
[master 921ae6b] p
 1 file changed, 0 insertions(+), 0 deletions(-)
 rename app.py => app.py.disabled (100%)
Counting objects: 3, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (2/2), done.
Writing objects: 100% (2/2), 224 bytes, done.
Total 2 (delta 1), reused 0 (delta 0)
remote: Stopping PYTHON cart
remote: running develop
remote: running egg_info
remote: writing requirements to YourAppName.egg-info/requires.txt
remote: writing YourAppName.egg-info/PKG-INFO
remote: writing top-level names to YourAppName.egg-info/top_level.txt
remote: writing dependency_links to YourAppName.egg-info/dependency_links.txt
remote: writing requirements to YourAppName.egg-info/requires.txt
remote: writing YourAppName.egg-info/PKG-INFO
remote: writing top-level names to YourAppName.egg-info/top_level.txt
remote: writing dependency_links to YourAppName.egg-info/dependency_links.txt
remote: warning: manifest_maker: standard file 'setup.py' not found
remote: 
remote: reading manifest file 'YourAppName.egg-info/SOURCES.txt'
remote: writing manifest file 'YourAppName.egg-info/SOURCES.txt'
remote: running build_ext
remote: Creating /var/lib/openshift/848108158837399257874432/python/virtenv/lib/python2.7/site-packages/YourAppName.egg-link (link to .)
remote: YourAppName 1.0 is already the active version in easy-install.pth
remote: 
remote: Installed /var/lib/openshift/848108158837399257874432/python/virtenv
remote: Processing dependencies for YourAppName==1.0
remote: Searching for gevent==0.13.8
remote: Best match: gevent 0.13.8
remote: Processing gevent-0.13.8-py2.7-linux-x86_64.egg
remote: gevent 0.13.8 is already the active version in easy-install.pth
remote: 
remote: Using /var/lib/openshift/848108158837399257874432/python/virtenv/lib/python2.7/site-packages/gevent-0.13.8-py2.7-linux-x86_64.egg
remote: Searching for greenlet==0.4.1
remote: Best match: greenlet 0.4.1
remote: Processing greenlet-0.4.1-py2.7-linux-x86_64.egg
remote: greenlet 0.4.1 is already the active version in easy-install.pth
remote: 
remote: Using /var/lib/openshift/848108158837399257874432/python/virtenv/lib/python2.7/site-packages/greenlet-0.4.1-py2.7-linux-x86_64.egg
remote: Finished processing dependencies for YourAppName==1.0
remote: Script /var/lib/openshift/848108158837399257874432/python//virtenv/bin/activate.csh cannot be made relative (it's not a normal script that starts with #!/var/lib/openshift/848108158837399257874432/python/virtenv/bin/python)
remote: Script /var/lib/openshift/848108158837399257874432/python//virtenv/bin/activate.fish cannot be made relative (it's not a normal script that starts with #!/var/lib/openshift/848108158837399257874432/python/virtenv/bin/python)
remote: Starting application cpy27
remote: Script /var/lib/openshift/848108158837399257874432/python//virtenv/bin/activate.csh cannot be made relative (it's not a normal script that starts with #!/var/lib/openshift/848108158837399257874432/python/virtenv/bin/python)
remote: Script /var/lib/openshift/848108158837399257874432/python//virtenv/bin/activate.fish cannot be made relative (it's not a normal script that starts with #!/var/lib/openshift/848108158837399257874432/python/virtenv/bin/python)
remote: Starting PYTHON cart
To ssh://848108158837399257874432.rhcloud.com/~/git/cpy27.git/
   b6e57bc..921ae6b  master -> master

4. Check this app's process
# rhc ssh cpy27 'ps -ef'
UID        PID  PPID  C STIME TTY          TIME CMD
503      24583     1  0 01:49 ?        00:00:00 /usr/sbin/httpd -C Include /var/lib/openshift/848108158837399257874432/python//etc/conf.d/*.conf -f /var/lib/openshift/848108158837399257874432/python//etc/conf/httpd_nolog.conf -k start
503      24584 24583  0 01:49 ?        00:00:00 /usr/sbin/rotatelogs /var/lib/openshift/848108158837399257874432/python//logs//error_log-%Y%m%d-%H%M%S-%Z 86400
503      24585 24583  0 01:49 ?        00:00:00 /usr/sbin/rotatelogs /var/lib/openshift/848108158837399257874432/python//logs//access_log-%Y%m%d-%H%M%S-%Z 86400
503      24586 24583  0 01:49 ?        00:00:00 /usr/sbin/httpd -C Include /var/lib/openshift/848108158837399257874432/python//etc/conf.d/*.conf -f /var/lib/openshift/848108158837399257874432/python//etc/conf/httpd_nolog.conf -k start
503      24587 24583  0 01:49 ?        00:00:00 /usr/sbin/httpd -C Include /var/lib/openshift/848108158837399257874432/python//etc/conf.d/*.conf -f /var/lib/openshift/848108158837399257874432/python//etc/conf/httpd_nolog.conf -k start
503      24588 24583  0 01:49 ?        00:00:00 /usr/sbin/httpd -C Include /var/lib/openshift/848108158837399257874432/python//etc/conf.d/*.conf -f /var/lib/openshift/848108158837399257874432/python//etc/conf/httpd_nolog.conf -k start
503      25134 25119  0 01:50 ?        00:00:00 sshd: 848108158837399257874432@notty
503      25135 25134  7 01:50 ?        00:00:00 ps -ef