+++ This bug was initially created as a clone of Bug #1012721 +++ Description of problem: When deploying a simple python cartridge in OSE 1.2.1 into an environment where there is NO internet access there is no way to point the application to an internal mirror. So, when i create an app and then edit it the first time (any simple edit) commit and push i get: > remote: Stopping PYTHON cart > remote: httpd (no pid file) not running > remote: setup.py found. Setting up virtualenv > remote: 'import site' failed; use -v for traceback > remote: New python executable in > /var/lib/openshift/523becb91c95a4918a000004/python//virtenv/bin/python > remote: Installing setuptools...................... > remote: Complete output from command > /var/lib/openshift/5...n/virtenv/bin/python -c "#!python > remote: \"\"\"Bootstra...sys.argv[1:]) > remote: > remote: > remote: > remote: > remote: > remote: > remote: " /usr/lib/python2.6/s...ols-0.6c11-py2.6.egg: > remote: 'import site' failed; use -v for traceback > remote: Downloading > http://pypi.python.org/packages/2.6/s/setuptools/setuptools-0.6c11-py2.6.egg > remote: Traceback (most recent call last): > remote: File "<string>", line 279, in <module> > remote: File "<string>", line 211, in main > remote: File "<string>", line 159, in download_setuptools > remote: File "/usr/lib64/python2.6/urllib2.py", line 126, in urlopen > remote: return _opener.open(url, data, timeout) > remote: File "/usr/lib64/python2.6/urllib2.py", line 391, in open > remote: response = self._open(req, data) > remote: File "/usr/lib64/python2.6/urllib2.py", line 409, in _open > remote: '_open', req) > remote: File "/usr/lib64/python2.6/urllib2.py", line 369, in _call_chain > remote: result = func(*args) > remote: File "/usr/lib64/python2.6/urllib2.py", line 1190, in http_open > remote: return self.do_open(httplib.HTTPConnection, req) > remote: File "/usr/lib64/python2.6/urllib2.py", line 1165, in do_open > remote: raise URLError(err) > remote: urllib2.URLError: <urlopen error [Errno 113] No route to host> > remote: ---------------------------------------- > remote: ...Installing setuptools...done. > remote: Traceback (most recent call last): > remote: File "/usr/bin/virtualenv", line 3, in <module> > remote: virtualenv.main() > remote: File "/usr/lib/python2.6/site-packages/virtualenv.py", line 928, in > main > remote: never_download=options.never_download) > remote: File "/usr/lib/python2.6/site-packages/virtualenv.py", line 1040, > in create_environment > remote: search_dirs=search_dirs, never_download=never_download) > remote: File "/usr/lib/python2.6/site-packages/virtualenv.py", line 593, in > install_setuptools > remote: search_dirs=search_dirs, never_download=never_download) > remote: File "/usr/lib/python2.6/site-packages/virtualenv.py", line 567, in > _install_req > remote: cwd=cwd) > remote: File "/usr/lib/python2.6/site-packages/virtualenv.py", line 1006, > in call_subprocess > remote: % (cmd_desc, proc.returncode)) > remote: OSError: Command /var/lib/openshift/5...n/virtenv/bin/python -c > "#!python > remote: \"\"\"Bootstra...sys.argv[1:]) > remote: > remote: > remote: > remote: > remote: > remote: > remote: " /usr/lib/python2.6/s...ols-0.6c11-py2.6.egg failed with error > code 1 > remote: An error occurred executing 'gear postreceive' > remote: > remote: For more details about the problem, try running the command again > with the '--trace' option. I edited the projects setup.py to use my internal URL: setup(name='YourAppName', version='1.0', description='OpenShift App', author='Your Name', author_email='example', url='http://internalpypi.me.org/sigs', # install_requires=['Django>=1.3'], ) But on next push the same error occurs and: http://pypi.python.org/packages/2.6/s/setuptools/setuptools-0.6c11-py2.6.egg is still used. Version-Release number of selected component (if applicable): How reproducible: every time Steps to Reproduce: 1. rhc create-app python1 python-2.6 2. edit a file (any simple edit to trigger a rebuild and restart) 3. edit setup.py to point at an internal mirror 4. commit 5. push Actual results: requests go against http://pypi.python.org/packages/2.6/s/setuptools/setuptools-0.6c11-py2.6.egg Expected results: requests go against http://internalpypi.me.org/.../setuptools-0.6c11-py2.6.egg Additional info: --- Additional comment from RHEL Product and Program Management on 2013-09-26 22:37:56 EDT --- Since this issue was entered in bugzilla, the release flag has been set to ? to ensure that it is properly evaluated for this release. --- Additional comment from Brenton Leanhardt on 2013-09-27 09:58:15 EDT --- Can the internal server be a proxy? Setting the HTTP_PROXY and HTTPS_PROXY environment variables worked as expected with setup.py. You could set them for all gears by creating /etc/openshift/env/HTTP{,S}_PROXY files with the content being the company's internal proxy. Otherwise it could likely be set in a pre_build action hook for only that gear. --- Additional comment from Mrunal Patel on 2013-09-27 12:44:50 EDT --- We set the python mirror in the build for python cartridge. We should allow deployments to customize the mirror by setting an enviroment variable under /etc/openshift/env and then make the python cartridge use that variable. We could name it as OPENSHIFT_PYPI_MIRROR_URL or something similar. Thanks, Mrunal --- Additional comment from Brenton Leanhardt on 2013-09-30 09:40:59 EDT --- Here's how to apply this: cd /var/lib/openshift/.cartridge_repository/redhat-python/0.0.1/ patch -p3 < /path/to/BZ1012721.patch The expected output is: patching file versions/2.6/bin/control patching file versions/shared/bin/build To undo this change you can apply the patch in reverse: cd /var/lib/openshift/.cartridge_repository/redhat-python/0.0.1/ patch -R -p3 < /path/to/BZ1012721.patch Once that is complete you can set /etc/openshift/env/OPENSHIFT_PYPI_MIRROR_URL. The content of that file should be a url to a PyPi index. --- Additional comment from Brenton Leanhardt on 2013-09-30 09:46:11 EDT --- Mrunal let me know what you think of the patch. I see no reason why we couldn't set OPENSHIFT_PYPI_MIRROR_URL for all Nodes in Online and avoid using the hardcoded rhcloud approach. --- Additional comment from Mrunal Patel on 2013-09-30 11:08:35 EDT --- Hi, Brenton, Yes, the patch looks good and makes sense to use for Online as well. Thanks, Mrunal --- Additional comment from Brenton Leanhardt on 2013-09-30 11:11:55 EDT --- August, let us know if this will solve the customer's problem and we'll upstream the patch and get it into OSE. --- Additional comment from August Simonelli on 2013-10-01 00:51:09 EDT --- This works well in my tests and would be of GREAT benefit! --- Additional comment from Brenton Leanhardt on 2013-10-01 08:02:42 EDT --- OK, I'll work to get this upstream and then backport to OSE. We should be able to release it with 1.2.4 in just a few weeks.
Commit pushed to master at https://github.com/openshift/origin-server https://github.com/openshift/origin-server/commit/1dc9f16db6b130a1c7792906bc86ef9e7f416087 Bug 1014339 - Support for setting $OPENSHIFT_PYPI_MIRROR_URL
Commit pushed to master at https://github.com/openshift/li https://github.com/openshift/li/commit/d64b3f32f819174361a935a7c83fe29c384f9f04 Bug 1014339 - Setting OPENSHIFT_PYPI_MIRROR_URL in rhc-node for all gears
For testing this I would suggest at least the following test cases: * edit /etc/hosts and making sure pypi.python.org resolves to something bogus. Do not set /etc/openshift/env/OPENSHIFT_PYPI_MIRROR_URL. This test should result in an error if setup.py is edited to include django. * leave the bad pypi.python.org DNS entry and then set /etc/openshift/env/OPENSHIFT_PYPI_MIRROR_URL. On a EC2 devenv you should be able to use http://mirror1.ops.rhcloud.com/mirror/python/web/simple. Including django in setup.py will result in using the mirror. * remove the bad /etc/hosts for pypi.python.org and typo the /etc/openshift/env/OPENSHIFT_PYPI_MIRROR_URL entry. In this case it should fall back to pypi.python.org. You can do this for both the python-2.6 and 3.3 to cover all code paths.
It's fixed, verified on devenv_3881, please refer to the following results: 1. Create a python app rhc create-app python2 python-2.6 2. Set OPENSHIFT_PYPI_MIRROR_URL to internal mirror rhc env set -e OPENSHIFT_PYPI_MIRROR_URL='http://internalpypi.me.org/sigs' -a python2 3. Disalbe external python mirror and include django cat setup.py from setuptools import setup setup(name='YourAppName', version='1.0', description='OpenShift App', author='Your Name', author_email='example', # url='http://www.python.org/sigs/distutils-sig/', install_requires=['Django>=1.3'], ) 4. Do some random changes and push touch test.txt git add . ; git commit -amp; git push [master efe9acb] p 1 file changed, 2 insertions(+), 2 deletions(-) Counting objects: 5, done. Delta compression using up to 4 threads. Compressing objects: 100% (3/3), done. Writing objects: 100% (3/3), 282 bytes, done. Total 3 (delta 2), reused 0 (delta 0) remote: Stopping PYTHON cart remote: [Thu Oct 10 06:25:31 2013] [warn] PassEnv variable SHELL was undefined remote: [Thu Oct 10 06:25:31 2013] [warn] PassEnv variable USER was undefined remote: [Thu Oct 10 06:25:31 2013] [warn] PassEnv variable LOGNAME was undefined remote: Waiting for stop to finish remote: Building git ref 'master', commit efe9acb remote: running develop remote: running egg_info remote: creating YourAppName.egg-info remote: writing requirements to YourAppName.egg-info/requires.txt remote: writing YourAppName.egg-info/PKG-INFO remote: writing top-level names to YourAppName.egg-info/top_level.txt remote: writing dependency_links to YourAppName.egg-info/dependency_links.txt remote: writing requirements to YourAppName.egg-info/requires.txt remote: writing YourAppName.egg-info/PKG-INFO remote: writing top-level names to YourAppName.egg-info/top_level.txt remote: writing dependency_links to YourAppName.egg-info/dependency_links.txt remote: writing manifest file 'YourAppName.egg-info/SOURCES.txt' remote: reading manifest file 'YourAppName.egg-info/SOURCES.txt' remote: writing manifest file 'YourAppName.egg-info/SOURCES.txt' remote: running build_ext remote: Creating /var/lib/openshift/52567fb634e7849fbc0002aa/app-deployments/2013-10-10_06-25-33.994/dependencies/virtenv/lib/python2.6/site-packages/YourAppName.egg-link (link to .) remote: Adding YourAppName 1.0 to easy-install.pth file remote: remote: Installed /var/lib/openshift/52567fb634e7849fbc0002aa/app-deployments/2013-10-10_06-25-33.994/repo remote: Processing dependencies for YourAppName==1.0 remote: Searching for Django>=1.3 remote: Reading https://pypi.python.org/simple/Django/ remote: Best match: Django 1.5.4 remote: Downloading https://pypi.python.org/packages/source/D/Django/Django-1.5.4.tar.gz#md5=b2685469bb4d1fbb091316e21f4108de remote: Processing Django-1.5.4.tar.gz remote: Writing /tmp/easy_install-t_ySS_/Django-1.5.4/setup.cfg remote: Running Django-1.5.4/setup.py -q bdist_egg --dist-dir /tmp/easy_install-t_ySS_/Django-1.5.4/egg-dist-tmp-rvq3uv remote: warning: no previously-included files matching '__pycache__' found under directory '*' remote: warning: no previously-included files matching '*.py[co]' found under directory '*' remote: zip_safe flag not set; analyzing archive contents... remote: django.test._doctest: module references __file__ remote: django.test._doctest: module MAY be using inspect.getsourcefile remote: django.utils.version: module references __file__ remote: django.utils.module_loading: module references __path__ remote: django.utils.autoreload: module references __file__ remote: django.utils.unittest.loader: module references __file__ remote: django.utils.unittest.collector: module references __file__ remote: django.utils.translation.trans_real: module references __file__ remote: django.views.i18n: module references __file__ remote: django.contrib.flatpages.tests.views: module references __file__ remote: django.contrib.flatpages.tests.middleware: module references __file__ remote: django.contrib.flatpages.tests.csrf: module references __file__ remote: django.contrib.flatpages.tests.templatetags: module references __file__ remote: django.contrib.gis.tests.geogapp.tests: module references __file__ remote: django.contrib.gis.tests.layermap.tests: module references __file__ remote: django.contrib.gis.tests.geo3d.tests: module references __file__ remote: django.contrib.gis.geometry.test_data: module references __file__ remote: django.contrib.admindocs.views: module references __file__ remote: django.contrib.admin.bin.compress: module references __file__ remote: django.contrib.auth.tests.context_processors: module references __file__ remote: django.contrib.auth.tests.views: module references __file__ remote: django.contrib.auth.tests.forms: module references __file__ remote: django.contrib.staticfiles.storage: module references __file__ remote: django.contrib.formtools.tests.__init__: module references __file__ remote: django.contrib.formtools.tests.wizard.namedwizardtests.tests: module references __file__ remote: django.contrib.formtools.tests.wizard.wizardtests.tests: module references __file__ remote: django.contrib.sitemaps.tests.http: module references __file__ remote: django.core.management.templates: module references __path__ remote: django.core.management.sql: module references __file__ remote: django.core.management.__init__: module references __file__ remote: django.core.management.__init__: module references __path__ remote: django.core.management.commands.makemessages: module references __file__ remote: django.core.management.commands.loaddata: module references __file__ remote: django.core.management.commands.loaddata: module references __path__ remote: django.db.utils: module references __file__ remote: django.db.models.loading: module references __file__ remote: django.template.loaders.app_directories: module references __file__ remote: Adding Django 1.5.4 to easy-install.pth file remote: Installing django-admin.py script to /var/lib/openshift/52567fb634e7849fbc0002aa/python/virtenv/bin remote: remote: Installed /var/lib/openshift/52567fb634e7849fbc0002aa/app-deployments/2013-10-10_06-25-33.994/dependencies/virtenv/lib/python2.6/site-packages/Django-1.5.4-py2.6.egg remote: Finished processing dependencies for YourAppName==1.0 remote: Script /var/lib/openshift/52567fb634e7849fbc0002aa/python//virtenv/bin/activate.csh cannot be made relative (it's not a normal script that starts with #!/var/lib/openshift/52567fb634e7849fbc0002aa/python/virtenv/bin/python) remote: Script /var/lib/openshift/52567fb634e7849fbc0002aa/python//virtenv/bin/activate.fish cannot be made relative (it's not a normal script that starts with #!/var/lib/openshift/52567fb634e7849fbc0002aa/python/virtenv/bin/python) remote: Preparing build for deployment remote: Prepared deployment artifacts in /var/lib/openshift/52567fb634e7849fbc0002aa/app-deployments/2013-10-10_06-25-33.994 remote: Deployment id is 2910bb8b remote: Activating deployment remote: Script /var/lib/openshift/52567fb634e7849fbc0002aa/python//virtenv/bin/activate.csh cannot be made relative (it's not a normal script that starts with #!/var/lib/openshift/52567fb634e7849fbc0002aa/python/virtenv/bin/python) remote: Script /var/lib/openshift/52567fb634e7849fbc0002aa/python//virtenv/bin/activate.fish cannot be made relative (it's not a normal script that starts with #!/var/lib/openshift/52567fb634e7849fbc0002aa/python/virtenv/bin/python) remote: Starting PYTHON cart remote: Result: success remote: Activation status: success remote: Deployment completed To ssh://52567fb634e7849fbc0002aa.rhcloud.com/~/git/python2.git/ b545f1f..efe9acb master -> master