My Python 2.7 application run cron like Python 2.6 I create a minutely cron than only run "python --version". Return 2.6 when the cartridge is 2.7. This happens since a two days, and block the cron execution. How reproducible: If I execute "python --version" through SSH shows 2.7.
Jhon, the issue here seems to be that rhcsh is not invoked so the LD_PATH, among other things, is not set, when running cron jobs in a user context. I see two ways to fix this: 1) we can invoke rhcsh from within the cron wrapper script cron_runjobs.sh 2) the code in oo-scheduled-jobs for daemon_as_user should be doing a better job setting up the environment. Thoughts on which is the "right" way to fix this? I'm inclined towards #2 since I suspect more than cron could be affected.
michal: seems like cron_runjobs.sh:45 isn't working properly if I write a cron script and dump the environment, this is what's in it: OPENSHIFT_PYTHON_LD_LIBRARY_PATH_ELEMENT=/opt/rh/python27/root/usr/lib64 LD_LIBRARY_PATH= it also seems like we'd want to invoke rhcsh to properly setup the path so that we pick up the python-2.7 binary first, just like the gear env does.
Ben: Thanks, this is indeed a bug, the LD_LIBRARY_PATH should be set to the python path. Working on it now.
Moises: The workaround for this would be to add this as a shebang: #!/bin/rhcsh (which should properly set all env vars).
Returns that /bin/rhcsh doesn't exists. I tried with #!/usr/bin/rhcsh as shebang, but returns: Mon Mar 17 12:07:19 EDT 2014: START minutely cron run __________________________________________________________________________ /var/lib/openshift/XXXXXX/app-root/runtime/repo//.openshift/cron/minutely/XXXX: /usr/bin/rhcsh: line 7: source: filename argument required source: usage: source filename [arguments] ********************************************************************* You are accessing a service that is for use only by authorized users. If you do not have authorization, discontinue use at once. Any use of the services is subject to the applicable terms of the agreement which can be found at: https://www.openshift.com/legal ********************************************************************* Welcome to OpenShift shell This shell will assist you in managing OpenShift applications. !!! IMPORTANT !!! IMPORTANT !!! IMPORTANT !!! Shell access is quite powerful and it is possible for you to accidentally damage your application. Proceed with care! If worse comes to worst, destroy your application with "rhc app delete" and recreate it !!! IMPORTANT !!! IMPORTANT !!! IMPORTANT !!! Type "help" for more info. WARNING: This ssh terminal was started without a tty. It is highly recommended to login with: ssh -t __________________________________________________________________________ Mon Mar 17 12:07:20 EDT 2014: END minutely cron run - status=0 __________________________________________________________________________ And not continues with the rest of the cron script. (In reply to Michal Fojtik from comment #4) > Moises: The workaround for this would be to add this as a shebang: > > #!/bin/rhcsh > > (which should properly set all env vars).
Moises, OK sorry for confusion here. The problem is really that the LD_LIBRARY_PATH is not set as Ben pointed out. The quickest workaround would be to copy the LD_LIBRARY_PATH from rhc ssh into your cron script and export it there. The proper fix is here: https://github.com/openshift/origin-server/pull/4976
Thanks, the workaround run, but need to export the PATH variable too.
Commit pushed to master at https://github.com/openshift/origin-server https://github.com/openshift/origin-server/commit/3755dfc6b3b53aee50d7904413d9f1d3168b2e60 Bug 1076626 - Fix LD_LIBRARY_PATH for cron_runjobs
Verified on devenv_4526, the version of python-2.7 displays like 2.7.5 now: [py27-dm.dev.rhcloud.com 5327f034f94cc54cc9000221]\> cat cron/log/cron.minutely.log __________________________________________________________________________ Tue Mar 18 04:21:05 EDT 2014: START minutely cron run __________________________________________________________________________ /var/lib/openshift/5327f034f94cc54cc9000221/app-root/runtime/repo//.openshift/cron/minutely/version.sh: Python 2.7.5 __________________________________________________________________________ Tue Mar 18 04:21:05 EDT 2014: END minutely cron run - status=0
Recheck other cartridges which has update in sprint41/origin_cartridge_31: python 3.3, ruby 1.9,nodejs 0.10, mysql-5.5 and postgresql-9.2 Both mysql-5.5 and postgresql-9.2 are failed, mysql-5.5: failed in rhcsh and cron job; postgresql-9.2: failed in cron job, but succeed in rhcsh: [sn10-dm.dev.rhcloud.com 53281b3cf94cc5e69d0001d4]\> mysql --version /usr/bin/mysql Ver 14.14 Distrib 5.1.73, for redhat-linux-gnu (x86_64) using readline 5.1 [sn10-dm.dev.rhcloud.com 53281b3cf94cc5e69d0001d4]\> cat cron/log/cron.minutely.log __________________________________________________________________________ Tue Mar 18 07:10:16 EDT 2014: START minutely cron run __________________________________________________________________________ /var/lib/openshift/53281b3cf94cc5e69d0001d4/app-root/runtime/repo//.openshift/cron/minutely/version.sh: mysql Ver 14.14 Distrib 5.1.73, for redhat-linux-gnu (x86_64) using readline 5.1 __________________________________________________________________________ Tue Mar 18 07:10:16 EDT 2014: END minutely cron run - status=0 __________________________________________________________________________ [sn10-dm.dev.rhcloud.com 53281b3cf94cc5e69d0001d4]\> psql --version psql (PostgreSQL) 9.2.7 [sn10-dm.dev.rhcloud.com 53281b3cf94cc5e69d0001d4]\> cat cron/log/cron.minutely.log __________________________________________________________________________ Tue Mar 18 07:16:21 EDT 2014: START minutely cron run __________________________________________________________________________ /var/lib/openshift/53281b3cf94cc5e69d0001d4/app-root/runtime/repo//.openshift/cron/minutely/version.sh: psql (PostgreSQL) 8.4.20 contains support for command-line editing __________________________________________________________________________ Tue Mar 18 07:16:21 EDT 2014: END minutely cron run - status=0
The 'mysql' command is not reporting the '5.5' due to a known limitation of MySQL 5.5 SCL. They just don't ship the client libraries there so we use the 'system' mysql command. That is why the version is 'incorrect'. So the mysql part is not a bug. The psql command is strange, I think it should report the correct version but I think you need to call it using 'scl enable' context. I think this is not a bug, but known limitation.
Just came across this bug as mentioned in StackOverflow, when trying to find out more information on failed cron jobs in Ruby 1.9. Seems similar enough for my Ruby 1.9 Cartridge. I ran a minutely cronjob with ruby --version and it shows ruby 1.8.7 (2011-06-30 patchlevel 352) [x86_64-linux]. Should I open a new bug or can just tag on here? Thanks!
Ok just went through the pull request in GitHub. Seems like a fix in the cron cartridge. I tried out the instructions given above, exporting the LD_LIBRARY_PATH and PATH in my cron script and I managed to get ruby 1.9.3p448 (2013-06-27) [x86_64-linux] for my Ruby 1.9 cartridge. Thanks!
Jason thanks for reporting this, this bug is related to all SCL based frameworks, so Ruby 1.9 is affected, same for NodeJS 0.10 The PR should fix for all the carts, I will rename this bug to be more clear.
Verified on devenv_4532, versions are correct when cron cartridge embeded now(except mysql which is a known limitation) [py33-dm.dev.rhcloud.com 532900bb0e6e4896af0000ac]\> cat cron/log/cron.minutely.log __________________________________________________________________________ Tue Mar 18 22:37:08 EDT 2014: START minutely cron run __________________________________________________________________________ /var/lib/openshift/532900bb0e6e4896af0000ac/app-root/runtime/repo//.openshift/cron/minutely/version.sh: Python 3.3.2 psql (PostgreSQL) 9.2.7 __________________________________________________________________________ Tue Mar 18 22:37:08 EDT 2014: END minutely cron run – status=0 Others display: Python 2.7.5 ruby 1.9.3p448 (2013-06-27) [x86_64-linux] PHP 5.4.16 Nodejs:v0.10.5 mysql Ver 14.14 Distrib 5.1.73, for redhat-linux-gnu (x86_64) using readline 5.1
Also verified on STG(devenv-stage_758) The versions are correct in rhcsh python-2.7: Python 2.7.5 python-3.3: Python 3.3.2 nodejs-0.10: v0.10.5 postgresql-9.2: psql (PostgreSQL) 9.2.7 ruby-1.9: ruby 1.9.3p448 (2013-06-27) [x86_64-linux] php-5.4: PHP 5.4.16 (cli) (built: Dec 6 2013 01:17:01) Copyright (c) 1997-2013 The PHP Group Zend Engine v2.4.0, Copyright (c) 1998-2013 Zend Technologies
I came across this bug report as my Python 3.3 & cron app is failing to run cron jobs. I am getting the following error when the cron job starts: python: error while loading shared libraries: libpython3.3m.so.1.0: cannot open shared object file: No such file or directory Is the same bug or something different? My cron jobs worked fine until Openshift update around March 13. I'm new to this and I see somebody posted a work around but I'm not sure exactly how to implement it. A more detailed explanation would be helpful. Any other ideas?
Hello fatfantasma, I did the below steps. Hope it helps you. 1. Logged in via SSH to the OpenShift app. 2. Ran the command 'echo $LD_LIBRARY_PATH' and 'echo $PATH' to get the current LD_LIBRARY_PATH and PATH variable values. 3. Added 2 lines of export into the cron script, like below. export LD_LIBRARY_PATH="/opt/rh/postgresql92/root/usr/lib64:/opt/rh/ruby193/root/usr/lib64" export PATH="/var/lib/openshift/<masked>/.gem/bin:/opt/rh/ruby193/root/usr/bin:/opt/rh/postgresql92/root/usr/bin:/bin:/usr/bin:/usr/sbin" 4. Add and deploy to OpenShift and the cron job should work. Thanks.
@Jason Thanks for the detailed fix. My cron jobs are working again :) However, I am still getting the following error in the cron log. I have no idea if it is related or not. Thanks again! python: error while loading shared libraries: libpython3.3m.so.1.0: cannot open shared object file: No such file or directory
(In reply to Jason Ng PT from comment #20) > Hello fatfantasma, > > I did the below steps. Hope it helps you. > > 1. Logged in via SSH to the OpenShift app. > 2. Ran the command 'echo $LD_LIBRARY_PATH' and 'echo $PATH' to get the > current LD_LIBRARY_PATH and PATH variable values. > 3. Added 2 lines of export into the cron script, like below. > > export > LD_LIBRARY_PATH="/opt/rh/postgresql92/root/usr/lib64:/opt/rh/ruby193/root/ > usr/lib64" > export > PATH="/var/lib/openshift/<masked>/.gem/bin:/opt/rh/ruby193/root/usr/bin:/opt/ > rh/postgresql92/root/usr/bin:/bin:/usr/bin:/usr/sbin" > > 4. Add and deploy to OpenShift and the cron job should work. > > Thanks. Thanks for the fix Jason Ng PT. It's working fine.