Bug 1014339

Summary:	no way to use an internal python mirror with python-2.6 cartridges
Product:	OKD	Reporter:	Brenton Leanhardt <bleanhar>
Component:	Containers	Assignee:	Brenton Leanhardt <bleanhar>
Status:	CLOSED CURRENTRELEASE	QA Contact:	libra bugs <libra-bugs>
Severity:	low	Docs Contact:
Priority:	unspecified
Version:	2.x	CC:	asimonel, bmeng, chunchen, libra-onpremise-devel, mpatel
Target Milestone:	---
Target Release:	---
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:	1012721	Environment:
Last Closed:	2013-10-17 13:26:23 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1012721

Description Brenton Leanhardt 2013-10-01 18:40:16 UTC

+++ This bug was initially created as a clone of Bug #1012721 +++

Description of problem:

When deploying a simple python cartridge in OSE 1.2.1 into an environment where there is NO internet access there is no way to point the application to an internal mirror. 

So, when i create an app and then edit it the first time (any simple edit) commit and push i get:

> remote: Stopping PYTHON cart
> remote: httpd (no pid file) not running
> remote: setup.py found. Setting up virtualenv
> remote: 'import site' failed; use -v for traceback
> remote: New python executable in
> /var/lib/openshift/523becb91c95a4918a000004/python//virtenv/bin/python
> remote: Installing setuptools......................
> remote: Complete output from command
> /var/lib/openshift/5...n/virtenv/bin/python -c "#!python
> remote: \"\"\"Bootstra...sys.argv[1:])
> remote:
> remote:
> remote:
> remote:
> remote:
> remote:
> remote: " /usr/lib/python2.6/s...ols-0.6c11-py2.6.egg:
> remote: 'import site' failed; use -v for traceback
> remote: Downloading
> http://pypi.python.org/packages/2.6/s/setuptools/setuptools-0.6c11-py2.6.egg
> remote: Traceback (most recent call last):
> remote: File "<string>", line 279, in <module>
> remote: File "<string>", line 211, in main
> remote: File "<string>", line 159, in download_setuptools
> remote: File "/usr/lib64/python2.6/urllib2.py", line 126, in urlopen
> remote: return _opener.open(url, data, timeout)
> remote: File "/usr/lib64/python2.6/urllib2.py", line 391, in open
> remote: response = self._open(req, data)
> remote: File "/usr/lib64/python2.6/urllib2.py", line 409, in _open
> remote: '_open', req)
> remote: File "/usr/lib64/python2.6/urllib2.py", line 369, in _call_chain
> remote: result = func(*args)
> remote: File "/usr/lib64/python2.6/urllib2.py", line 1190, in http_open
> remote: return self.do_open(httplib.HTTPConnection, req)
> remote: File "/usr/lib64/python2.6/urllib2.py", line 1165, in do_open
> remote: raise URLError(err)
> remote: urllib2.URLError: <urlopen error [Errno 113] No route to host>
> remote: ----------------------------------------
> remote: ...Installing setuptools...done.
> remote: Traceback (most recent call last):
> remote: File "/usr/bin/virtualenv", line 3, in <module>
> remote: virtualenv.main()
> remote: File "/usr/lib/python2.6/site-packages/virtualenv.py", line 928, in
> main
> remote: never_download=options.never_download)
> remote: File "/usr/lib/python2.6/site-packages/virtualenv.py", line 1040,
> in create_environment
> remote: search_dirs=search_dirs, never_download=never_download)
> remote: File "/usr/lib/python2.6/site-packages/virtualenv.py", line 593, in
> install_setuptools
> remote: search_dirs=search_dirs, never_download=never_download)
> remote: File "/usr/lib/python2.6/site-packages/virtualenv.py", line 567, in
> _install_req
> remote: cwd=cwd)
> remote: File "/usr/lib/python2.6/site-packages/virtualenv.py", line 1006,
> in call_subprocess
> remote: % (cmd_desc, proc.returncode))
> remote: OSError: Command /var/lib/openshift/5...n/virtenv/bin/python -c
> "#!python
> remote: \"\"\"Bootstra...sys.argv[1:])
> remote:
> remote:
> remote:
> remote:
> remote:
> remote:
> remote: " /usr/lib/python2.6/s...ols-0.6c11-py2.6.egg failed with error
> code 1
> remote: An error occurred executing 'gear postreceive'
> remote:
> remote: For more details about the problem, try running the command again
> with the '--trace' option.

I edited the projects setup.py to use my internal URL:

setup(name='YourAppName',
       version='1.0',
       description='OpenShift App',
       author='Your Name',
       author_email='example',
       url='http://internalpypi.me.org/sigs',
#      install_requires=['Django>=1.3'],
      )

But on next push the same error occurs and:

http://pypi.python.org/packages/2.6/s/setuptools/setuptools-0.6c11-py2.6.egg

is still used.

Version-Release number of selected component (if applicable):


How reproducible:

every time

Steps to Reproduce:
1. rhc create-app python1 python-2.6
2. edit a file (any simple edit to trigger a rebuild and restart)
3. edit setup.py to point at an internal mirror
4. commit
5. push

Actual results:

requests go against http://pypi.python.org/packages/2.6/s/setuptools/setuptools-0.6c11-py2.6.egg

Expected results:

requests go against

http://internalpypi.me.org/.../setuptools-0.6c11-py2.6.egg


Additional info:

--- Additional comment from RHEL Product and Program Management on 2013-09-26 22:37:56 EDT ---

Since this issue was entered in bugzilla, the release flag has been
set to ? to ensure that it is properly evaluated for this release.

--- Additional comment from Brenton Leanhardt on 2013-09-27 09:58:15 EDT ---

Can the internal server be a proxy?  Setting the HTTP_PROXY and HTTPS_PROXY environment variables worked as expected with setup.py.

You could set them for all gears by creating /etc/openshift/env/HTTP{,S}_PROXY files with the content being the company's internal proxy.  Otherwise it could likely be set in a pre_build action hook for only that gear.

--- Additional comment from Mrunal Patel on 2013-09-27 12:44:50 EDT ---

We set the python mirror in the build for python cartridge. We should
allow deployments to customize the mirror by setting an enviroment variable
under /etc/openshift/env and then make the python cartridge use that variable.

We could name it as OPENSHIFT_PYPI_MIRROR_URL or something similar.

Thanks,
Mrunal

--- Additional comment from Brenton Leanhardt on 2013-09-30 09:40:59 EDT ---

Here's how to apply this:

cd /var/lib/openshift/.cartridge_repository/redhat-python/0.0.1/
patch -p3 < /path/to/BZ1012721.patch

The expected output is:

patching file versions/2.6/bin/control
patching file versions/shared/bin/build

To undo this change you can apply the patch in reverse:

cd /var/lib/openshift/.cartridge_repository/redhat-python/0.0.1/
patch -R -p3 < /path/to/BZ1012721.patch

Once that is complete you can set /etc/openshift/env/OPENSHIFT_PYPI_MIRROR_URL.  The content of that file should be a url to a PyPi index.

--- Additional comment from Brenton Leanhardt on 2013-09-30 09:46:11 EDT ---

Mrunal let me know what you think of the patch.  I see no reason why we couldn't set OPENSHIFT_PYPI_MIRROR_URL for all Nodes in Online and avoid using the hardcoded rhcloud approach.

--- Additional comment from Mrunal Patel on 2013-09-30 11:08:35 EDT ---

Hi, Brenton,
Yes, the patch looks good and makes sense to use for Online as well.

Thanks,
Mrunal

--- Additional comment from Brenton Leanhardt on 2013-09-30 11:11:55 EDT ---

August, let us know if this will solve the customer's problem and we'll upstream the patch and get it into OSE.

--- Additional comment from August Simonelli on 2013-10-01 00:51:09 EDT ---

This works well in my tests and would be of GREAT benefit!

--- Additional comment from Brenton Leanhardt on 2013-10-01 08:02:42 EDT ---

OK, I'll work to get this upstream and then backport to OSE.  We should be able to release it with 1.2.4 in just a few weeks.

Comment 1 openshift-github-bot 2013-10-03 14:08:10 UTC

Commit pushed to master at https://github.com/openshift/origin-server

https://github.com/openshift/origin-server/commit/1dc9f16db6b130a1c7792906bc86ef9e7f416087
Bug 1014339 - Support for setting $OPENSHIFT_PYPI_MIRROR_URL

Comment 2 openshift-github-bot 2013-10-03 14:08:16 UTC

Commit pushed to master at https://github.com/openshift/li

https://github.com/openshift/li/commit/d64b3f32f819174361a935a7c83fe29c384f9f04
Bug 1014339 - Setting OPENSHIFT_PYPI_MIRROR_URL in rhc-node for all gears

Comment 3 Brenton Leanhardt 2013-10-03 15:17:19 UTC

For testing this I would suggest at least the following test cases:

* edit /etc/hosts and making sure pypi.python.org resolves to something bogus.  Do not set /etc/openshift/env/OPENSHIFT_PYPI_MIRROR_URL.  This test should result in an error if setup.py is edited to include django.

* leave the bad pypi.python.org DNS entry and then set /etc/openshift/env/OPENSHIFT_PYPI_MIRROR_URL. On a EC2 devenv you should be able to use http://mirror1.ops.rhcloud.com/mirror/python/web/simple.  Including django in setup.py will result in using the mirror.

* remove the bad /etc/hosts for pypi.python.org and typo the /etc/openshift/env/OPENSHIFT_PYPI_MIRROR_URL entry.  In this case it should fall back to pypi.python.org.

You can do this for both the python-2.6 and 3.3 to cover all code paths.

Comment 4 chunchen 2013-10-10 10:51:36 UTC

It's fixed, verified on devenv_3881, please refer to the following results:

1. Create a python app
rhc create-app python2 python-2.6
2. Set OPENSHIFT_PYPI_MIRROR_URL to internal mirror
rhc env set -e OPENSHIFT_PYPI_MIRROR_URL='http://internalpypi.me.org/sigs' -a python2
3. Disalbe  external python mirror and include django
cat setup.py
from setuptools import setup

setup(name='YourAppName',
      version='1.0',
      description='OpenShift App',
      author='Your Name',
      author_email='example',
#      url='http://www.python.org/sigs/distutils-sig/',
      install_requires=['Django>=1.3'],
     )
4. Do some random changes and push
touch test.txt
git add . ; git commit -amp; git push

[master efe9acb] p
 1 file changed, 2 insertions(+), 2 deletions(-)
Counting objects: 5, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 282 bytes, done.
Total 3 (delta 2), reused 0 (delta 0)
remote: Stopping PYTHON cart
remote: [Thu Oct 10 06:25:31 2013] [warn] PassEnv variable SHELL was undefined
remote: [Thu Oct 10 06:25:31 2013] [warn] PassEnv variable USER was undefined
remote: [Thu Oct 10 06:25:31 2013] [warn] PassEnv variable LOGNAME was undefined
remote: Waiting for stop to finish
remote: Building git ref 'master', commit efe9acb
remote: running develop
remote: running egg_info
remote: creating YourAppName.egg-info
remote: writing requirements to YourAppName.egg-info/requires.txt
remote: writing YourAppName.egg-info/PKG-INFO
remote: writing top-level names to YourAppName.egg-info/top_level.txt
remote: writing dependency_links to YourAppName.egg-info/dependency_links.txt
remote: writing requirements to YourAppName.egg-info/requires.txt
remote: writing YourAppName.egg-info/PKG-INFO
remote: writing top-level names to YourAppName.egg-info/top_level.txt
remote: writing dependency_links to YourAppName.egg-info/dependency_links.txt
remote: writing manifest file 'YourAppName.egg-info/SOURCES.txt'
remote: reading manifest file 'YourAppName.egg-info/SOURCES.txt'
remote: writing manifest file 'YourAppName.egg-info/SOURCES.txt'
remote: running build_ext
remote: Creating /var/lib/openshift/52567fb634e7849fbc0002aa/app-deployments/2013-10-10_06-25-33.994/dependencies/virtenv/lib/python2.6/site-packages/YourAppName.egg-link (link to .)
remote: Adding YourAppName 1.0 to easy-install.pth file
remote: 
remote: Installed /var/lib/openshift/52567fb634e7849fbc0002aa/app-deployments/2013-10-10_06-25-33.994/repo
remote: Processing dependencies for YourAppName==1.0
remote: Searching for Django>=1.3
remote: Reading https://pypi.python.org/simple/Django/
remote: Best match: Django 1.5.4
remote: Downloading https://pypi.python.org/packages/source/D/Django/Django-1.5.4.tar.gz#md5=b2685469bb4d1fbb091316e21f4108de
remote: Processing Django-1.5.4.tar.gz
remote: Writing /tmp/easy_install-t_ySS_/Django-1.5.4/setup.cfg
remote: Running Django-1.5.4/setup.py -q bdist_egg --dist-dir /tmp/easy_install-t_ySS_/Django-1.5.4/egg-dist-tmp-rvq3uv
remote: warning: no previously-included files matching '__pycache__' found under directory '*'
remote: warning: no previously-included files matching '*.py[co]' found under directory '*'
remote: zip_safe flag not set; analyzing archive contents...
remote: django.test._doctest: module references __file__
remote: django.test._doctest: module MAY be using inspect.getsourcefile
remote: django.utils.version: module references __file__
remote: django.utils.module_loading: module references __path__
remote: django.utils.autoreload: module references __file__
remote: django.utils.unittest.loader: module references __file__
remote: django.utils.unittest.collector: module references __file__
remote: django.utils.translation.trans_real: module references __file__
remote: django.views.i18n: module references __file__
remote: django.contrib.flatpages.tests.views: module references __file__
remote: django.contrib.flatpages.tests.middleware: module references __file__
remote: django.contrib.flatpages.tests.csrf: module references __file__
remote: django.contrib.flatpages.tests.templatetags: module references __file__
remote: django.contrib.gis.tests.geogapp.tests: module references __file__
remote: django.contrib.gis.tests.layermap.tests: module references __file__
remote: django.contrib.gis.tests.geo3d.tests: module references __file__
remote: django.contrib.gis.geometry.test_data: module references __file__
remote: django.contrib.admindocs.views: module references __file__
remote: django.contrib.admin.bin.compress: module references __file__
remote: django.contrib.auth.tests.context_processors: module references __file__
remote: django.contrib.auth.tests.views: module references __file__
remote: django.contrib.auth.tests.forms: module references __file__
remote: django.contrib.staticfiles.storage: module references __file__
remote: django.contrib.formtools.tests.__init__: module references __file__
remote: django.contrib.formtools.tests.wizard.namedwizardtests.tests: module references __file__
remote: django.contrib.formtools.tests.wizard.wizardtests.tests: module references __file__
remote: django.contrib.sitemaps.tests.http: module references __file__
remote: django.core.management.templates: module references __path__
remote: django.core.management.sql: module references __file__
remote: django.core.management.__init__: module references __file__
remote: django.core.management.__init__: module references __path__
remote: django.core.management.commands.makemessages: module references __file__
remote: django.core.management.commands.loaddata: module references __file__
remote: django.core.management.commands.loaddata: module references __path__
remote: django.db.utils: module references __file__
remote: django.db.models.loading: module references __file__
remote: django.template.loaders.app_directories: module references __file__
remote: Adding Django 1.5.4 to easy-install.pth file
remote: Installing django-admin.py script to /var/lib/openshift/52567fb634e7849fbc0002aa/python/virtenv/bin
remote: 
remote: Installed /var/lib/openshift/52567fb634e7849fbc0002aa/app-deployments/2013-10-10_06-25-33.994/dependencies/virtenv/lib/python2.6/site-packages/Django-1.5.4-py2.6.egg
remote: Finished processing dependencies for YourAppName==1.0
remote: Script /var/lib/openshift/52567fb634e7849fbc0002aa/python//virtenv/bin/activate.csh cannot be made relative (it's not a normal script that starts with #!/var/lib/openshift/52567fb634e7849fbc0002aa/python/virtenv/bin/python)
remote: Script /var/lib/openshift/52567fb634e7849fbc0002aa/python//virtenv/bin/activate.fish cannot be made relative (it's not a normal script that starts with #!/var/lib/openshift/52567fb634e7849fbc0002aa/python/virtenv/bin/python)
remote: Preparing build for deployment
remote: Prepared deployment artifacts in /var/lib/openshift/52567fb634e7849fbc0002aa/app-deployments/2013-10-10_06-25-33.994
remote: Deployment id is 2910bb8b
remote: Activating deployment
remote: Script /var/lib/openshift/52567fb634e7849fbc0002aa/python//virtenv/bin/activate.csh cannot be made relative (it's not a normal script that starts with #!/var/lib/openshift/52567fb634e7849fbc0002aa/python/virtenv/bin/python)
remote: Script /var/lib/openshift/52567fb634e7849fbc0002aa/python//virtenv/bin/activate.fish cannot be made relative (it's not a normal script that starts with #!/var/lib/openshift/52567fb634e7849fbc0002aa/python/virtenv/bin/python)
remote: Starting PYTHON cart
remote: Result: success
remote: Activation status: success
remote: Deployment completed
To ssh://52567fb634e7849fbc0002aa.rhcloud.com/~/git/python2.git/
   b545f1f..efe9acb  master -> master