Bug 1012721 - no way to use an internal python mirror with python-2.6 cartridges in OSE
no way to use an internal python mirror with python-2.6 cartridges in OSE
Status: CLOSED ERRATA
Product: OpenShift Container Platform
Classification: Red Hat
Component: Containers (Show other bugs)
1.2.1
x86_64 Linux
unspecified Severity high
: ---
: ---
Assigned To: Brenton Leanhardt
libra bugs
:
Depends On: 1014339
Blocks:
  Show dependency treegraph
 
Reported: 2013-09-26 22:32 EDT by August Simonelli
Modified: 2017-03-08 12 EST (History)
8 users (show)

See Also:
Fixed In Version: openshift-origin-cartridge-python-0.4.8.2-1.el6op
Doc Type: Enhancement
Doc Text:
Previously in OpenShift Enterprise there was no way of using an internal Python mirror with Python-2.6 cartridges where internet access was not available. The Python cartridge can now have an internal mirror enabled by using the $OPENSHIFT_PYPI_MIRROR_URL setting. Changing this setting in /etc/openshift/env/OPENSHIFT_PYPI_MIRROR_URL on a node host is recommended as this will make it available for all gears. However, it can be set in a gear's action hook if desired.
Story Points: ---
Clone Of:
: 1014339 (view as bug list)
Environment:
Last Closed: 2013-10-16 11:39:41 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
support for setting OPENSHIFT_PYPI_MIRROR_URL (1.90 KB, patch)
2013-09-30 09:40 EDT, Brenton Leanhardt
no flags Details | Diff

  None (edit)
Description August Simonelli 2013-09-26 22:32:29 EDT
Description of problem:

When deploying a simple python cartridge in OSE 1.2.1 into an environment where there is NO internet access there is no way to point the application to an internal mirror. 

So, when i create an app and then edit it the first time (any simple edit) commit and push i get:

> remote: Stopping PYTHON cart
> remote: httpd (no pid file) not running
> remote: setup.py found. Setting up virtualenv
> remote: 'import site' failed; use -v for traceback
> remote: New python executable in
> /var/lib/openshift/523becb91c95a4918a000004/python//virtenv/bin/python
> remote: Installing setuptools......................
> remote: Complete output from command
> /var/lib/openshift/5...n/virtenv/bin/python -c "#!python
> remote: \"\"\"Bootstra...sys.argv[1:])
> remote:
> remote:
> remote:
> remote:
> remote:
> remote:
> remote: " /usr/lib/python2.6/s...ols-0.6c11-py2.6.egg:
> remote: 'import site' failed; use -v for traceback
> remote: Downloading
> http://pypi.python.org/packages/2.6/s/setuptools/setuptools-0.6c11-py2.6.egg
> remote: Traceback (most recent call last):
> remote: File "<string>", line 279, in <module>
> remote: File "<string>", line 211, in main
> remote: File "<string>", line 159, in download_setuptools
> remote: File "/usr/lib64/python2.6/urllib2.py", line 126, in urlopen
> remote: return _opener.open(url, data, timeout)
> remote: File "/usr/lib64/python2.6/urllib2.py", line 391, in open
> remote: response = self._open(req, data)
> remote: File "/usr/lib64/python2.6/urllib2.py", line 409, in _open
> remote: '_open', req)
> remote: File "/usr/lib64/python2.6/urllib2.py", line 369, in _call_chain
> remote: result = func(*args)
> remote: File "/usr/lib64/python2.6/urllib2.py", line 1190, in http_open
> remote: return self.do_open(httplib.HTTPConnection, req)
> remote: File "/usr/lib64/python2.6/urllib2.py", line 1165, in do_open
> remote: raise URLError(err)
> remote: urllib2.URLError: <urlopen error [Errno 113] No route to host>
> remote: ----------------------------------------
> remote: ...Installing setuptools...done.
> remote: Traceback (most recent call last):
> remote: File "/usr/bin/virtualenv", line 3, in <module>
> remote: virtualenv.main()
> remote: File "/usr/lib/python2.6/site-packages/virtualenv.py", line 928, in
> main
> remote: never_download=options.never_download)
> remote: File "/usr/lib/python2.6/site-packages/virtualenv.py", line 1040,
> in create_environment
> remote: search_dirs=search_dirs, never_download=never_download)
> remote: File "/usr/lib/python2.6/site-packages/virtualenv.py", line 593, in
> install_setuptools
> remote: search_dirs=search_dirs, never_download=never_download)
> remote: File "/usr/lib/python2.6/site-packages/virtualenv.py", line 567, in
> _install_req
> remote: cwd=cwd)
> remote: File "/usr/lib/python2.6/site-packages/virtualenv.py", line 1006,
> in call_subprocess
> remote: % (cmd_desc, proc.returncode))
> remote: OSError: Command /var/lib/openshift/5...n/virtenv/bin/python -c
> "#!python
> remote: \"\"\"Bootstra...sys.argv[1:])
> remote:
> remote:
> remote:
> remote:
> remote:
> remote:
> remote: " /usr/lib/python2.6/s...ols-0.6c11-py2.6.egg failed with error
> code 1
> remote: An error occurred executing 'gear postreceive'
> remote:
> remote: For more details about the problem, try running the command again
> with the '--trace' option.

I edited the projects setup.py to use my internal URL:

setup(name='YourAppName',
       version='1.0',
       description='OpenShift App',
       author='Your Name',
       author_email='example@example.com',
       url='http://internalpypi.me.org/sigs',
#      install_requires=['Django>=1.3'],
      )

But on next push the same error occurs and:

http://pypi.python.org/packages/2.6/s/setuptools/setuptools-0.6c11-py2.6.egg

is still used.

Version-Release number of selected component (if applicable):


How reproducible:

every time

Steps to Reproduce:
1. rhc create-app python1 python-2.6
2. edit a file (any simple edit to trigger a rebuild and restart)
3. edit setup.py to point at an internal mirror
4. commit
5. push

Actual results:

requests go against http://pypi.python.org/packages/2.6/s/setuptools/setuptools-0.6c11-py2.6.egg

Expected results:

requests go against

http://internalpypi.me.org/.../setuptools-0.6c11-py2.6.egg


Additional info:
Comment 2 Brenton Leanhardt 2013-09-27 09:58:15 EDT
Can the internal server be a proxy?  Setting the HTTP_PROXY and HTTPS_PROXY environment variables worked as expected with setup.py.

You could set them for all gears by creating /etc/openshift/env/HTTP{,S}_PROXY files with the content being the company's internal proxy.  Otherwise it could likely be set in a pre_build action hook for only that gear.
Comment 3 Mrunal Patel 2013-09-27 12:44:50 EDT
We set the python mirror in the build for python cartridge. We should
allow deployments to customize the mirror by setting an enviroment variable
under /etc/openshift/env and then make the python cartridge use that variable.

We could name it as OPENSHIFT_PYPI_MIRROR_URL or something similar.

Thanks,
Mrunal
Comment 4 Brenton Leanhardt 2013-09-30 09:40:59 EDT
Created attachment 805214 [details]
support for setting OPENSHIFT_PYPI_MIRROR_URL

Here's how to apply this:

cd /var/lib/openshift/.cartridge_repository/redhat-python/0.0.1/
patch -p3 < /path/to/BZ1012721.patch

The expected output is:

patching file versions/2.6/bin/control
patching file versions/shared/bin/build

To undo this change you can apply the patch in reverse:

cd /var/lib/openshift/.cartridge_repository/redhat-python/0.0.1/
patch -R -p3 < /path/to/BZ1012721.patch

Once that is complete you can set /etc/openshift/env/OPENSHIFT_PYPI_MIRROR_URL.  The content of that file should be a url to a PyPi index.
Comment 5 Brenton Leanhardt 2013-09-30 09:46:11 EDT
Mrunal let me know what you think of the patch.  I see no reason why we couldn't set OPENSHIFT_PYPI_MIRROR_URL for all Nodes in Online and avoid using the hardcoded rhcloud approach.
Comment 6 Mrunal Patel 2013-09-30 11:08:35 EDT
Hi, Brenton,
Yes, the patch looks good and makes sense to use for Online as well.

Thanks,
Mrunal
Comment 7 Brenton Leanhardt 2013-09-30 11:11:55 EDT
August, let us know if this will solve the customer's problem and we'll upstream the patch and get it into OSE.
Comment 8 August Simonelli 2013-10-01 00:51:09 EDT
This works well in my tests and would be of GREAT benefit!
Comment 9 Brenton Leanhardt 2013-10-01 08:02:42 EDT
OK, I'll work to get this upstream and then backport to OSE.  We should be able to release it with 1.2.4 in just a few weeks.
Comment 10 Brenton Leanhardt 2013-10-03 11:18:18 EDT
See the test cases for the upstream bug on how to test this.
Comment 12 Ma xiaoqiang 2013-10-08 04:25:49 EDT
I have verified it on puddle  [1.2/2013-10-03.1]:

1.create a python app
#rhc app create pythonapp python-2.6
2.set OPENSHIFT_PYPI_MIRROR_URL on Node
#echo http://mirror1.ops.rhcloud.com/mirror/python/web/simple >>/etc/openshift/env/OPENSHIFT_PYPI_MIRROR_URL
3.configure setup.py file
#setup.py
from setuptools import setup

setup(name='YourAppName',
      version='1.0',
      description='OpenShift App',
      author='Your Name',
      author_email='example@example.com',
      url='http://mirror1.ops.rhcloud.com/mirror/python/web/simple',
      install_requires=['Django==1.4.2'],
     )
4.git push

Output:
remote: Reading http://mirror1.ops.rhcloud.com/mirror/python/web/simple/Django/
remote: Best match: Django 1.4.2
remote: Downloading http://mirror1.ops.rhcloud.com/mirror/python/web/packages/source/D/Django/Django-1.4.2.tar.gz#md5=6ffecdc01ad360e1abdca1015ae0893a
remote: Processing Django-1.4.2.tar.gz
remote: Running Django-1.4.2/setup.py -q bdist_egg --dist-dir /tmp/easy_install-UPFOML/Django-1.4.2/egg-dist-tmp-1ITGbx
Comment 16 errata-xmlrpc 2013-10-16 11:39:41 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-1435.html

Note You need to log in before you can comment on or make changes to this bug.